Python: Manipulate string or binary bytes with StringIO

November 25th, 2008 mysurface

Sometimes it is not convenient to construct string using equal (=) like this:

str = "Hello, "
...
str = str + "my name is "
...
str = str + Name
print str

In python, we have string stream (StringIO) that will behave like file stream, you can construct your string like this:

str=StringIO()
...
str.write("Hello, ")
...
str.write("my name is ")
...
str.write(Name)
print str.getvalue()

The same way, you can construct your binary bytes with StringIO and write it into file once you are done.

bin=StringIO()
bin.write("/x5F/x5F%c" % 0xFF)
...
file = open ("my.bin","wb")
file.write(bin.getvalue())
file.close()

Posted in Developer, python | Hits: 10775 | 3 Comments »

python: writing binary file

November 20th, 2008 mysurface

Python is the best scripting language which I found it out perform Bash script as well as Lua. I like python’s scripting syntax, which is make sense and very convenient in string manipulations. Its now come to binary format manipulation, how convenient is python?

I have carry out some research on python into writing binary files. Either writing byte by byte manually, construct a binary stream then write it one short, or convert it from hex string, they are easy to write in python.

To open a file for binary writing is easy, it is the same way you do for reading, just change the mode into “wb”.

file = open("test.bin","wb")

But, how to write the binary byte into the file?
You may write it straight away with hex code like this:

file.write("\x5F\x9D\x3E")
file.close()

Now, check it out with hexedit,

hexedit test.bin

You will see this:

00000000   5F 9D 3E                                                                                                _.>
00000020
00000040

Now, open the file to append more bytes:

file = open("test.bin","ab")

What if I want to store by bin value into a stream and write it one short?

s ="\x45\xF3"
s = s + "%c%c" % (0x45,0xF3)
file.write(s)
file.close()

Any convenient ways if I can obtained a hex string, and want to convert it back to binary format?
Yes, you just need to import binascii

import binascii
hs="5B7F888489FEDA"
hb=binascii.a2b_hex(hs)
file.write(hb)
file.close()

Isn’t it easy? I love python. You may want to discover more functions of binascii with ipython.

Posted in python | Hits: 11341 | 1 Comment »

Bash script that process input from pipeline or file redirection

November 15th, 2008 mysurface

I remember I wrote a post regarding how to makes python processing string from the pipeline stream. This time, I find the needs to do it in bash script.

I created a hex string splitter script for my friend who needs to decode the raw data that will be written in hex string such as “5F443D95FEA3D4787AEDC4″. Every time, he split them byte by byte manually into this form (“5F 44 3D 95 FE A3 D4 78 7A ED C4″) for better reading. He was complain that the manual process was always bothering him. I told him that can be done by writting a simple bash script.

My hexsplit.sh will do this:

./hexsplit.sh "5F443D95FEA3D4787AEDC4"

But what if he grep this hex string from other file? Or what if the hex string are stored in a file? I want my bash script to be able to process hex string from pipeline stream as well as file redirection too, like this:

 grep "hex" dummy.log | cut -d"]" -f2 | ./hexsplit.sh

or like this:

 ./hexsplit.sh < dummy.log

How to write this hexsplit.sh?

hexsplit.sh

#!/bin/bash

if [ -e $1 ] ;then read str; else str=$1;fi
len=`expr length $str`
for (( a=0; a<=$len; a=a+2 )); do echo -n ${str:a:2}" "; done
echo ""

The important line is the if statement. If there are no argument specified ( -e $1), I read from the stream, else i take it from $1 ( param 1). It is so simple isn’t it?

Have fun!

Posted in Bash, pipeline | Hits: 12979 | 2 Comments »

List command line history with timestamp

October 16th, 2008 mysurface

History is a common command for shell to list out all the executed commands. It is very useful when it comes to investigation on what commands was executed that tear down the server. With the help of last command, you be able to track the login time of particular user as well as the the duration of the time he/she stays login.

last
...
mysurface    tty7         :0               Mon Oct  6 20:07 - down   (00:00)
reboot   system boot  2.6.24.4-64.fc8  Mon Oct  6 20:06          (00:00)
mysurface    pts/8        10.168.28.44     Mon Oct  6 17:42 - down   (01:58)
mysurface    pts/7        :0.0             Mon Oct  6 17:41 - 19:40  (01:59)
mysurface    pts/6        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 17:27 - 19:40  (02:13)
mysurface    pts/5        :0.0             Mon Oct  6 15:52 - 15:59  (00:07)
...

If the command line history could provides the date time of the commands being executed, that may really narrow down the scope of the user actions that cause the server malfunction. By default, history do not append with timestamp, but it is easy to configure it to display timestamp, you just need to set one environment variable HISTTIMEFORMAT.

HISTTIMEFORMAT takes format string of strftime. Check out the strftime manual to choose and construct the timestamp that suit your taste. My favorite is “%F %T “.

export HISTTIMEFORMAT="%F %T "

Execute history again and you will see the effect on the spot, bare in mind that the timestamp for command lines that executed at previous sessions may not valid, as the time was not tracked.

...
  994  2008-10-16 02:27:40 exit
  995  2008-10-16 01:12:20 iptables -nL
  996  2008-10-16 01:47:46 vi .bash_profile
  997  2008-10-16 01:47:55 history
  998  2008-10-16 01:48:03 . .bash_profile
  999  2008-10-16 01:48:04 history
 1000  2008-10-16 01:48:09 exit
 1001  2008-10-16 02:27:43 history
...

I would suggest you to put the export into ~/.bash_profile as well as /root/.bash_profile. In case you do not have .bash_profile, you can choose to put into ~/.bashrc.

Don’t mess up my servers! Your actions will be track!

Posted in Common, history, last | Hits: 20868 | 2 Comments »

XSLT processor command line

September 7th, 2008 mysurface

XSLT stand for XSL transformation, and XSL is EXtensible Stylesheet Language, it is XML-based Stylesheet Language defined by W3C. XSLT is one of the famous XML technology, XML that uses XSL needs transformation before that make sense. Some application such as firefox browser do support XSLT, with xsl source specified, it will be transform automatically.

If you have XSL, and xml that uses XSL, you can also transform by using a command call xsltproc. xsltproc is a tool from libxslt. The usage is straight forward:

xsltproc mytemp.xsl mytarget.xml

The line above will dump the transformed xml to standard output, which you can redirect it to any files.

With -o, the transformed xml will be dumped into file.

xsltproc mytemp.xsl mytarget.xml -o myresult.xml

p.s. For xsltproc, the targeted xml do not need to specified the xsl inline such as:

<?xml-stylesheet href="mytemp.xsl" type="text/xsl" ?>

To know more about XSLT check out W3C XSLT tutorial, or check out ways of transform XML with XSLT. You can also get some samples from the W3C tutorial to try out xsltproc.

Posted in Misc | Hits: 30533 | No Comments »

KDE based Fluxbox open folder tips

August 30th, 2008 mysurface

I am a KDE based Fluxbox user, I am face an issue that some of my gnome applications that have feature of allowing me to locate and open folder doesn’t really work as expected. Those gnome applications are gnome-do and picasa. When I open a folder from gnome-do or picasa, it will be open in firefox. Instead, I want it to open at konqueror.

I didn’t find any solution for that, due to the fact there are not much KDE based Fluxbox users around. But, at last I figure out a work around.

I realized that those gnome applications will call xdg-open when it is asked to open the folders. xdg-open is a bash script that will checks for global environment variable $DE and decide what file browser to execute. I have extracted the important part for reference:

if [ x"$DE" = x"" ]; then
    # if BROWSER variable is not set, check some well known browsers instead
    if [ x"$BROWSER" = x"" ]; then
        BROWSER=htmlview:firefox:mozilla:netscape:links:lynx
    fi
    DE=generic
fi

case "$DE" in
    kde)
    open_kde "$url"
    ;;

    gnome)
    open_gnome "$url"
    ;;

    xfce)
    open_xfce "$url"
    ;;

    generic)
    open_generic "$url"
    ;;

    *)
    exit_failure_operation_impossible "no method available for opening '$url'"
    ;;
esac

The workaround is assign $DE with value “kde”, edit your ~/.bash_profile or ~/.bashrc with:

export DE=kde

Relogin again and it is done.

Posted in fluxbox, picasa | Hits: 32656 | No Comments »

Experiencing with vmware server and Sun xVM VirtualBox

August 17th, 2008 mysurface

Recently I have tried Sun xVM VirtualBox 1.6.4 , I have compare vbox with Vmware server 1.0.4. After couple days of testing, its time for me to share some personal findings towards them. I may not able to provide graphs, and accurate figures to shows their performance and specification, but what I do is to only share my point of view as a layman, and how I like or dislike them based on the user experience.

First of all, let me list down my host machine specs:
Dell Vostro 1400 with Intel Core 2 Duo CPU T5270 @ 1.40GHz, 2G Ram.
I use 512 Mb Ram to load my guest os, I tried loading up myrinix live CD, as well as installed Windows XP.
Uses Fedora 8, my WM uses Fluxbox 1.0.0.

Before looking into performance and features, let us first look at the screenshot! I am loading up myrinix live cd at both virtual tool. The reason I choose myrinix because this great Live CD pre install with vbox guest additions as well as vmware tools, so that I can enjoy the mouse integration to move my mouse IN and OUT the guest os.

Virtual Box have separate window for virtual os settings and guest OS itself.

Vmware Server makes every Guest OSes in MDI, separate each guest oses into tabs, and the guest os settings in sidebar, NEAT!

Read the rest of this entry »

Posted in Misc, VirtualBox, vmware | Hits: 39978 | 3 Comments »

Virtualize your operating system with qemu

August 2nd, 2008 mysurface

QEMU is a processor emulator, it allows you to run variety of unmodified guest operating systems such as Linux, Windows, Solaris, Dos etc just like Vmware and VirtualBox. In fact VirtualBox dynamic recompiler are based on QEMU. QEMU supports emulating several hardware platforms, including x86, AMD64, Alpha, MIPS, ARM, PPC, SPARC etc.

QEMU comes with command line interface, where you pass your configurations by parameters, but there are 3rd party GUI front end, such as Qemulator and Qemu-Launcher. To improves the performance of QEMU, we needs either KQEMU kernel module or KVM. Both of them are accelerator that to increase the execution speed of QEMU.

In this post, I covers installation of QEMU as well as KQEMU. As well as simple example of the use of QEMU command lines, how to create image files that allows to execute by both QEMU as well as Vmware.

Read the rest of this entry »

Posted in qemu, qemu-img, vmware | Hits: 43713 | 5 Comments »

Syslog: Sending log from remote servers to syslog daemon

July 23rd, 2008 mysurface

syslog is a standard for logging service in Linux, it usually run as daemon like syslogd or rsyslogd. Syslog daemon will be forward and store logs in /var/log directory, you may configure it to store at separate location if you want. (we will look into it later). And there is a major file that store majority of logs, which is messages. Therefore, you may want to monitor linux messages logs by tailing /var/log/message.

Logs entry may come from various services, applications, kernel as well as remote servers if you enabled your syslog daemon to accept remote logs submissions. Syslog protocol is now standardized within the Syslog working group of the IETF, and it is been defines in RFC 3164.

Rsyslog is an enhanced multi-threaded syslogd with a focus on security and reliability. I think newer linux distro already replace syslogd with rsyslogd. For more information you can check out the wikipedia.

In this post, I briefly explain the facility and log levels of syslog protocol, how to configure syslogd as well as rsyslogd to accept logs from remote and also how to send logs remotely.
Read the rest of this entry »

Posted in Admin, logger, nc, rsyslogd, syslogd, tail | Hits: 49734 | No Comments »

Top Ten Processes Watcher

July 13th, 2008 mysurface

top command provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel. But if you want get something more specific, you must play some tricks on it. For example, I want a clean view of top ten busiest processes every seconds without those summary info, how should I do with top?

Batch mode operation
top is a real-time application that keeps display processes info sorted by cpu usage, if you redirects the outputs to a file like command line below, you will having your results mixed with ugly symbols, which is not what you want.

top > top.txt

Well, top support batch mode. By specified top on batch mode, top will display the outputs without arrange the views properly, but if you redirects it to a file, you won’t get those messy symbols mixed up.

top -b > top.txt

I just want to monitor top ten processes every seconds!
You can limits your views of top ten using head and tail, but this will STOP top from continuing to monitor the processes.

top -d 1 | head -n 17 | tail -n 11

Fortunately, we have watch to help, but you needs to put top in batch mode, and also you can ask top to stop after done displaying the result for one time. This will increase the accuracy of the result.

watch -n 1 "top -b -n 1 | head -n 17 | tail -n 11"

Posted in head, tail, top, watch | Hits: 49862 | No Comments »