Find common data from three files

Given 3 files, each with names of thousands of customers who bought something from Amazon.com on that particular day. Find the common customers (i.e. common names in file1 and file2 and file3)

Solution:

1) assuming UNIX like systems:

cat f1 f2 | sort | uniq -d > tmpfile;

cat f3 tmpfile | sort | uniq -d

-------------------
Better options:
sort f1 f2 f3 |uniq -d

http://www.softpanorama.org/Tools/Sort/unix_sort_examples_collection.shtml
sort -t ':' -k 3,4 /etc/passwd

http://amazoninterview.blogspot.com/2007/05/find-common-data-from-three-files.html

find files with phone number

with following two formats
(xxx) xxx-xxxx and xxx-xxx-xxxx.

-l list only filename
grep -l -R --perl-regexp "\b(\(\d{3}\)\s*|\d{3}-)\d{3}-\d{4}\b" * > output.txt

(xxx) or xxx-
\(\d{3}\)\s*|\d{3}-
xxx-xxxx
\d{3}-\d{4}


http://www.wellho.net/regex/grep.html
http://www.robelle.com/smugbook/regexpr.html
(grep doesn't support "+" "|" "()"
(xxx) or xxx-
egrep "\([0-9]\{3\})|[0-9]\{3\}-" ./*
ls -1 list 1 file one line:
ls -1 | egrep 'DIV_(pharm|dental|vision)

perl replace "search" with "replace" and make the file as backup
perl -pi.bak -e's/searchs/replaces/' file1 file2 file3 fileN

Using aircrack-ng to hack WPA wireless protocol

Forum
http://forum.aircrack-ng.org/index.php?PHPSESSID=175264d2aaf388739350642fae99a34a&board=1.0

Windows network commands
http://whirlpool.net.au/wiki/?tag=Windows_NW_Diag_Cmds



linux manual
http://www.aircrack-ng.org/doku.php?id=getting_started
http://www.aircrack-ng.org/doku.php?id=airserv-ng



airserv-ng -d "commview.dll| {my adapter id}"
此时airserv-ng就在正常载入驱动后，同时开始监听本地的666端口。换句话说，airserv-ng提供的是该无线网卡的网络服务，其他计算机上的用户也可以连接到这个端口来使用这块网卡，如图5-30所示。


步骤2：现在可以使用airodump-ng来搜索当前无线环境了。注意，要另开启一个CMD，再输入如下命令：

airodump-ng 127.0.0.1：666
这里在IP地址处输入为本机即127.0.0.1，端口采用的是默认的666。

airodump-ng --channel number -w filename 127.0.0.1:666
这里输入“airodump-ng --channel 7 -w onewpa 127.0.0.1:666”，回车后可看到如图5-32所示的内容。


步骤3：现在，就可以和前面在BackTrack 2中讲述的一样进行Deauth攻击了，另开启一个CMD，输入（如图5-33所示）：

aireplay-ng -0 1 -a AP’s MAC 127.0.0.1:666

在个别情况下，可能会在上面命令结束时出现wi_write<>:Operation now progress这样的提示，这个提示在很多无线网卡下都会出现，意思是当前进程调用被占用时，Deauth攻击在大多数情况下并不会受此影响。


步骤4：再开启一个CMD，输入命令查看是否捕获到WPA握手数据包。命令如下：

aircrack-ng 捕获的数据包名
如果没有捕获到WPA握手数据包，就会有显示的“0 handshake”，这个时候切回到刚才aireplay-ng所在的CMD中
重复进行Deauth攻击。

攻击过程中注意观察airodump-ng工作界面，当右上角出现如图5-35所示的提示时，就表示成功截获到WPA握手数据包了。


此时再次使用aircrack-ng 打开捕获的数据包，就可以看到截获到WPA <1 handshake>的显示了，
接下来便可以进行WPA本地破解了。


破解WPA-PSK输入命令如下：

aircrack-ng -w password.txt onewpa*.cap
这里介绍一个小技巧，可以在输入数据包名后面加一个*号，来自动将捕获的该名称的所有数据包整合导入。回车后，显示如图5-37所示。


如同在Linux下破解WPA一般耐心地等待，就可以看到WPA密钥成功解出了.

{1,5, -5, -8,2, -1,15 } -> {-5, -8, -1, 1, 5, 2,15}

每遇到一个负数, instead of exchange with the
first 正数，把前面的正数right round shift就可以了。

Index 0 1 2 3
Data 1 5 4 -5

L = 0, R = 3
right round shift from 1 to -5 就变成
-5 1 5 4

void shiftright(int* A, int L, int R) {
        int tmp=A[R],i;
        for (i=R; i>L; i--) {
                A[i] = A[i-1];
        }
        A[L] = tmp;
}


int main(){
        int a[] = {-7, -8, 9, -6, -11, -3, -5, 2, 3};
        int n, i, j= 0;
        n = sizeof(a)/sizeof(int);
        printf("array size %d\n", n);

        for(i=0; i

Software development under Linux

Posted on 2003, a little bit old, but still very useful guide for
linux beginners.
http://reallylongword.org/gentoo/


Quick start on python
http://reallylongword.org/python/


How to create SVN repository
http://reallylongword.org/svn/

grep/cut/perl usage examples

grep -o
Only print the match part.
grep -A num
Print num lines after the match

grep "\<" match the empty string at the beginning
of the word
$ echo "so happy i am " | grep -o "\"
happy


echo -e "\t"
Show special chars in script string.

cut -s
do not print lines not containing delimiters

#!/bin/bash
for file in $(ls *.html)
do
        match=$(perl parse.pl $file | grep -A 1   Manufacturer:)
        manu=$(echo $match | grep -w -o  Manufacturer:.*Mfr | sed 's/Manufacturer://g'|  sed 's/M
fr//g')
        date=$(echo $match | grep -w -o  "Date:.*NHTSA CAMPAIGN"| sed 's/Date://g'|  sed 's/NHTSA
 CAMPAIGN//g')
        comp=$(echo $match | grep -w -o  Component:.*Potential | sed 's/Component: //g'|  sed 's/
Potential//g')
        num=$(echo $match | grep -w -o  Affected:.*Summary | sed 's/Affected: //g'|  sed 's/Summa
ry//g')

        echo -e  "$manu\t$comp\t$num\t$date" >> table
done

#       file1="${file%.html}.txt"
#       echo $file1
#       manu=$(grep "Manufacturer:" $newfile| cut -d":" -s -f 2)
#       date=$(grep "ReportDate:"  $newfile| cut -b 19-32)
#       date=$(grep "ReportDate:"  $newfile| cut -b 19-32)
#       comp=$(grep "Component:" $newfile|sed 's/Component:/\t/g' |)

awk vs sed

"Doing even seemingly simple thing in sed can quickly result in large,
difficult expressions, making most usage of sed hard to understand. sed
works well for simple cases of string filtering, but awk can often be a
more maintainable choice."



http://www.osnews.com/story/21004/Awk_and_Sed_One-Liners_Explained
seq 19 | tr '\n' ,
tr translate \n with ,



http://reallylongword.org/sedawk/
"&" stands for the matched part of the text

sed 's!.*/!!'

[jwang@osnet1 bin ]$ echo "/users/abc/bla1.exe" |sed 's!.*/!!'

bla1.exe


Remove MSDOS carriage return characters of each line.
sed 's/\r$//'

#!/bin/bash

while read line
do
#   path=${line%%/*}  
   path=${line%/*} 
   echo $path
done < $1 > outfile

exit 0


#newusers/defgh/ik/albg2.exe
#->
#newusers/defgh/ik


#if %%  result is :
#newusers

linux npviewer problem

64-bit Firefox npwrapper (npviewer) plugin allows the 32-bit version
of Flash Player run in 64-bit Firefox. But it lock up all the time and
eating CPU cycles all the time.

How to kill npviewer process:
for name in $(ps ux | awk '/npviewer.bin/ && !/awk/ {print $2}'); do kill "$name"; done


Install firefox plugin: noscript or flashblock will solve the problem.


Download 64 bit flashplayer here:
sudo yum install flash-plugin libflashsupport
http://www.cyberciti.biz/tips/linux-install-flash-player-10.html
wget http://fpdownload.macromedia.com/get/flashplayer/current/install_flash_player_10_linux.tar.gz
extract and copy libflashplayer.so to /usr/lib/mozilla/plugins/libflashplayer.so

Notes:
Seems the 64 bit player still not work right now for firefox
http://labs.adobe.com/downloads/flashplayer10.html

download search result from cgi/cfm form

get second column and sort and remove duplicate

cat FLAT_RCL.txt | cut -f 2 | grep '^0[2-9][A-X]' | sort|uniq > dest.txt


You can use command line browser tools to download file
lynx -dump $url > file.txt


But Most importantly you can use curl tools.
Suppose there is a search toolbar like the following,
you want to download all the results from the
query.

NHTSA Campaign ID Number:
(e.g. 01V166002)

You can use
curl -d "SearchType=QuickSearch&rcl_id=$x&summary=true&Search=Search" http://www-odi.nhtsa.dot.gov/recalls/results.cfm > $x.html

The most important is the rcl_id you provided (as a parameter here).

For more information about curl, refer to
http://curl.haxx.se/docs/manual.html




How to readline from file:

N=0
cat ./dest.txt |  while read LINE;
do
N=$((N+1))
x=$LINE
cd ~/fay
curl -d "SearchType=QuickSearch&rcl_id=$x&summary=true&Search=Search" http://www-odi.nhtsa.dot.gov/recalls/results.cfm > 
$x.html
done

TCP forwarding/ssh tunneling

zz from kartik mail
Here's a way to use ssh tunneling from outside department to avoid
logging in twice to reach osnet1.

Most of you should have an account on alpha.cs.binghamton.edu

We can tunnel our ssh connection to
osnet1.cs.binghamton.edu via alpha.cs.binghamton.edu
as follows

Assuming you are using UNIX there, first establish a tunnel
to osnet1 as follows:

$ ssh -f -L 2222:osnet1.cs.binghamton.edu:22 alpha.cs.binghamton.edu sleep 24h

When it asks for password, type your password
for alpha.cs.binghamton.edu.

This tells ssh to forward your local port 2222 via
alpha.cs.binghamton.edu to port 22 on
osnet1.cs.binghamton.edu

The -f places the above ssh command in background.
The "sleep 24h" just executes the sleep command for
a long time (24 hrs  -- you can increase it to whatever you want)

Next, ssh to your local port 2222 as if you were doing an ssh to osnet1

$ ssh -p 2222 localhost

When it asks for password again, type your password for osnet1.

If you want to transfer files between your local machine and osnet1 then

$ scp -P 2222  localhost:

(note the capital P) .... and so on.

This should log you into osnet1. From osnet1 you can ssh to any
other machine in the lab.

If you set up you ssh keys properly on both alpha.cs and osnet1.cs
then you don't even have to type your password each time. Just place
the above two commands in a script and it'll seamlessly log you in.
(I can guide you through this if you don't know how).

Here's some more explanation of TCP forwarding using SSH

http://www.gsp.com/cgi-bin/man.cgi?section=1&topic=ssh#5

rename and replace

Replace parm1 with parm2 within each file in the directory

#!/usr/bin/env bash
FILE_LIST="`ls *.html`" 
echo FILE_LIST: ${FILE_LIST}

for f in ${FILE_LIST}
do
        sed 's/$1/$2/g' f> TMPFILE && 
                    mv TMPFILE f
done

echo "replace Script done"

Cut usage:
http://www.thelinuxblog.com/using-cut-shellscript-string-manipulation/

bingsun2% echo "daemon:x:1:1:daemon:/usr/sbin:/bin/sh" | cut -d: -f1,5
daemon:daemon

//To batch rename jan09 to 09jan
//`` is to force sh to execute as command
for i in $(ls ~/fay)
do
a=`echo $i | cut -c1-3`
b=`echo $i | cut -c4-5`
mv $i $b$a
done

resize hvm guest

$ dd if=/dev/zero of=new_image bs=1 count=new_size_of_ext3_in_bytes
$ dd if=old_image of=new_image conv=notrunc #copy the old image bytes to
the new image (without overwritting the new file)
$ losetup -o 32256 /dev/loop0 new_image # /root doesn't actually live
at front of image for HVM
$ fsck -f /dev/loop0 #check the new image for errors
$ echo "new_image_bytes / 8225280" | bc # this is number of cylinders
in new device
$ fdisk new_image
$ # go to the "extra" menu and type "c" and type the new number of cylinders
$ # go back to main menu
$ # delete primary partition (yes, delete it)
$ # add new primary partition w/ defaults # resize partition to size of
actual image file
$ # 'w' write out fdisk and exit
$ resize2fs /dev/loop0 20G #resize ext3 to whatever GB you want MINUS
a gigabyte ..... just to be on the safe side
$ losetup -d /dev/loop0 # get rid of the loop device
$ edit xen config and test new image (don't delete the old one!)

wget example

The following script
convert windows accept filenames
convert links to local link
convert cgi/jsp link to html file name
recursive download 3 levels

wget  --restrict-file-names=windows -E -r  -p  -l 3  -k   -Ahtml -Ahtm -Ajpg -np
 -nv -nd -nH http://www.fda.gov/Safety/Recalls/ArchiveRecalls/2007/default.htm

The following example want to download only html file from
one website and put it all the files in one certain directory
and convert link relationship to make it cross-referable.

#!/usr/bin/env bash

MONTH=("jan" "feb" "mar" "apr" "may" "jun" "jul" "aug" "sep" "oct" "nov" "dec")
YEAR=("00" "01" "02" "03" "04" "05" "06" "07" "08" "09" "90" "91" "92" "93" "94"
 "95" "96" "97" "98" "99")


for mon in ${MONTH[*]}
do
    for year in ${YEAR[*]}
    do
        x=$mon$year
 mkdir ~/fay/$x
        cd ~/fay/
 wget  -p -r -l 2  -k -nc -Ahtml  -P$x  -np -nv -nd -nH  http://www.cpsc.
gov/cpscpub/prerel/prerel$x.html
    done
done

echo "Wj Script done"

wget useful options

#usefule options

To download a single page and all its requisites (even if they exist
on separate websites), and make sure the lot displays properly
locally, this author likes to use a few options in addition to -p:

        wget -E -H -k -K -p http:///

-np     no parent
-H
       --span-hosts
           Enable spanning across hosts when doing recursive retrieving.

-A.c download .c file
-E   add html extension to cgi/jsp script links
-K
       --backup-converted
           When converting a file, back up the original version with a .orig
           suffix.  Affects the behavior of -N.
-k     
       --convert-links   The same as the above, but convert the links in the HTML
         files to point to local files, so you can view the
         documents off-line:

                 wget --convert-links -r http://www.gnu.org/ -o gnulog
---restrict-file-names=mode
           windows doesn't accept certain file names like <>?
           it can replace with certain chars like ? to #

-nc
    --noclobber(no overwrite) 
           With -nc, download the same file will overwritten the old. 
           With -r -nc, download the same file will avoid overwriting, just use the original copy. 
           When running Wget without -N, -nc, -r, or p, downloading the same
           file in the same directory will result in the original copy of file
           being preserved and the second copy being named file.1.
 -F
       --force-html
           When input is read from a file, force it to be treated as an HTML
           file.  This enables you to retrieve relative links from existing
           HTML files on your local disk, by adding some url to
           HTML, or using the --base command-line option.
-p
       --page-requisites
           This option causes Wget to download all the files that are
           necessary to properly display a given HTML page.  This includes
           such things as inlined images, sounds, and referenced stylesheets.

 -q
       --quiet
           Turn off Wget’s output.

 -nv
       --no-verbose
           Turn off verbose without being completely quiet (use -q for that),
           which means that error messages and basic information still get
           printed.

wget -r -p -l 2 -np -nc -q  http://www.cpsc.gov/cpscpub/prerel/prerel.html

12球称重问题

脑筋问题挺好玩的。

有12个乒乓球,其中有一个重量不一样,但不知是轻是重,
请用天平称三次,把坏球找出来解法
----------------------------------

A1A2A3A4
B1B2B3B4
C1C2C3C4


A1-4 vs B1-4

• 如果A=B

那坏球在C堆里

C12 vs A12，如果相等，坏球在C34里，C3 vs A1 ....
如果不等，坏球在C12里，C1 vs A1 ...

• 如果A<B(A>B同样处理，switch AB即可)

A1B1C1 vs A2B2B3

1. A1B1C1 = A2B2B3

A3A4, B4, A3B4 vs C1C2

• A3B4 vs C1C2,
• if 相等，==》A4轻

• if < ==》A3 轻
  
• > ==》B4重

2. A1B1C1 < A2B2B3
不可能B1轻或者A2重，所以或者A1 轻，或者是B2B3重，

• A1B2 vs C1C2
  

3.A1B1C1 > A2B2B3
either B1 重或者A2轻，

• B1 vs C1 ....

思路关键就是要记住第一步的轻重对解决问题有指导性作用。

-----------------------------
还有8球问题，比较简单，分成332，先比较33就可以了。
这一题好像要知道坏球是重还是轻才行。

Multikernel

Cache coherence become a problem with the increase of cores.

Using message passing instead of shared memory can solve the problem.

http://docs.google.com/present/view?id=dc3nqpnt_19hsg376hs

try to explain slide 21,

Broadcast vs Multicast:

TLB shootdown – the process of maintaining TLB consistency
by invalidating entries when pages are unmapped
– is one of the simplest operations in a multiprocessor

OS that requires global coordination. It is also
a short, but latency-critical, operation and so represents
a worst-case comparison for a multikernel.

Broadcast performs badly due to the cache-coherence protocol used
by AMD64 processors. When the master updates the line, it is invalidated
in all other caches. Each slave core then pulls the new copy from the master’s
cache.

Multicast protocol, the master sends a URPC message to the first core of each processor,

which forwards it to the other three cores in the package.
Since they all share an L3 cache, this second message is much cheaper.

IP Loopback

Table 4 shows that Barrelfish achieves higher throughput,
fewer cache misses, and lower interconnect utilization,
particularly in the reverse direction from sink to
source. This occurs because sending packets as URPC
messages avoids any shared-memory other than the
URPC channel and packet payload; conversely, Linux
causes more cache-coherence traffic for shared-memory
synchronization.
Barrelfish also benefits by avoiding kernel crossings.

AICT new problem

I got into two problems right now with AICT
AICT Performance not as good as expected.

1.

S CPU1 T CPU2
I get the same performance(7300tran/s) with raise_softirq to T.
with aict I cannot get any performance improvement.

Since at CPU 2 side , T with C1 and C2 are competing for
CPU resources, with aict, T should get some kind of priority
hence smaller


2.
If I give up CPU at Target side right after target finish its I/O
shedule() doesn't take effect as I expected.
Problem: xenloop doesn't know when target should give up
cpu during xmit_packets().

Task: to find after schedule(), when could S get schduled

AICT debug printk would help

I stuck with a problem for 2 weeks. Just found out that basic printk would give me
why it happens.
Problem description:
Start 2 VM would panic at insert_vcpu BUG_ON
complaining VCPU already in the runqueue.

Just printk at
insert_vcpu to see whether it is aict context
if (per_cpu(schedule_data, cpu).aict_source)
printk("source domid= %d \n", per_cpu(schedule_data, cpu).aict_source->domain->domain_id);
if (per_cpu(schedule_data, cpu).aict_target)
printk("target domid= %d \n", per_cpu(schedule_data, cpu).aict_target->domain->domain_id);

Should take the lesson and review the problem next time.

interview questions

e) 一个有10个指针的数组，该指针是指向一个整型数的（An array of 10 pointers to integers）
f) 一个指向有10个整型数数组的指针（A pointer to an array of 10 integers）
g) 一个指向函数的指针，该函数有一个整型参数并返回一个整型数（A pointer to a function that takes an integer as an argument and returns an integer）
h) 一个有10个指针的数组，该指针指向一个函数，该函数有一个整型参数并返回一个整型数（ An array of ten pointers to functions that take an integer argument and return an integer ）

答案是：
e) int *a[10]; // An array of 10 pointers to integers
f) int (*a)[10]; // A pointer to an array of 10 integers
g) int (*a)(int); // A pointer to a function a that takes an integer argument and returns an integer
h) int (*a[10])(int); // An array of 10 pointers to functions that take an integer argument and return an integer

#include 
#include 

int main(void) {


 typedef union {long i; int k[5]; char c;} DATE;
 struct date { int cat; DATE cow; double dog;} too;
 DATE max;
 int b[5];
 printf("struct %d\n", sizeof(struct date) );
 printf("union %d\n", sizeof(DATE));
 printf("double %d\n", sizeof(double));
 printf("long %d\n", sizeof(long));
 printf("array int %d\n", sizeof(b));
 //actual result is  40 24 8 8 20
}

sizeof
long 8, double 8, long long 8, short 2
64位机上测试的结果:
Summary: struct and union has common point, 大小永远为里面最大类型(max=8)的倍数,
struct 按照最大类型的大小来判断是否需要补齐。

install KVM using QEMU

Insert kvm.ko && (kvm-amd.ko || kvm-intel.ko)

Install qemu-kvm

Creating a vdisk

qemu-img create -f qcow2 vdisk.img 10G

Booting from ISO

qemu-system-x86_64 -hda vdisk.img -cdrom /vmroot/ftp/ubuntu-9.04-desktop-amd64.iso -boot d -m 384 -usbdevice tablet -vnc :1

where -boot [a|c|d|n] boot from floppy (a), hard disk (c), CD-ROM (d), or network (n)
-m is RAM size in MBs

Follow the normal OS installation procedure.

To get the network working

#in file /etc/network/interfaces

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet static
address 10.128.0.20
netmask 255.255.0.0
broadcast 10.128.255.255
gateway 10.128.0.1
bridge_ports eth0
bridge_fd 0
bridge_maxwait 0
bridge_stp off

Creating a qtap device and adding it to the bridge

sudo modprobe tun
sudo tunctl -b -u root -t qtap0
sudo brctl addif br0 qtap0
sudo ifconfig qtap0 up 0.0.0.0 promisc

Now we will have network ready by configuring /etc/network/interfaces in the guest.

Booting up the VM (We will call the first machine as source and the machine where we are going to migrate the VM as destination)
Make sure you have vncserver installed on both source and destination machines.

qemu-system-x86_64 -hda vdisk.img -net nic,macaddr=52:54:00:39:81:49 -net tap,ifname=qtap0,script=no,downscript=no -vnc :1 -usbdevice tablet (On source machine)
vncviewer :5901 (On your desktop)

To migrate the VM
On target side execute following command to start the blank window.

qemu-system-x86_64 -hda vdisk.img -net nic,macaddr=52:54:00:39:81:49 -net tap,ifname=qtap0,script=no,downscript=no -incoming tcp:0:4444 -vnc :1 -usbdevice tablet

press Ctrl+Alt+2 to switch to qemu console and execute following command to migrate the VM.

migrate -d tcp::

After the migration we can connect to 5901 port of the destination host through vncviewer by executing following command on your desktop machine.

vncviewer :5901

C++ static variable and static function

1.static variable can only be declared inside class
cannot initialized inside class
and it must be initialized after class.

2. static func can only access static variable
other functions inside class can access static variable freely

3. static variable can be private/public
when it is private, it cannnot be accessed via
Base::var = 0;

4.A static member function cannot have access to the 'this'
pointer of the class

5.Do not declare a static member function as virtual!!!