Turning up the Big data curtain!

Its been a long time since we have posted. There is a reason for that. Please keep reading.
Change is inevitable. Let your glorious past be cherished by others. Let it remain in your memories. Let it add value to your profession. Let it give confidence to you. But do not let it ruin your future plans.

Technology is the prime. One can stand out from the crowd only by showing the logical decision makings. As per NASSCOM, 40% of IT professionals are unskilled and unprepared to deal with upcoming IT models. My view is, out of these 40% there are 30% who knows about this but still showing no dissent towards it. People keep discussing it and losing time very fast.

Its already late if we have not upgraded ourselves yet. With a lot of study we have done about next-gen technologies, we could figure out that the IT industry is running towards automation and the jobs are on stake. Time to pulll up socks and take guard.

Based on several trends, we could reach to the following things:

1. Big Data and analytics related technologies are need of hour. Its time to analyze trends and make predictive models.
2. Virtualization is mandatory in Big companies for internal infrastructure or even for products point of view.
3. Virtualization is via Cloud now. Be it Infrastructure as a service (IAAS), Software as a service (SAAS), Platform as a service (PAAS), Data as a service (DAAS) etc.

This is just a beginning.

I personally want to make a one year plan for myself with expertise in a particular order.

Big Data and Hadoop --> Map Reduce --> Data Analytics using various tools --> Data Science --> Machine Learning -> Robotics for automation

Its measurable and time bounded and of course one year of dedicated hard work. But this is how we do for ourselves.

Yes, there can be more points and many more points but above is what which we have studied. Now need to take the things in our own way. Lets start fresh.

This blog will be more about Big data and other stuff from now onwards. We have got great testimonials from Unix/Linux professionals about our blog but now we hope to keep up with the pace of next-gen technologies. We are starting with Big data. It will be as it comes basis. The more we read, the more we post. The more queries we get, the more we answer. Lets make it a combined effort to engage ourselves in some self investment for our own benefits.

Keep studying, you are in IT!

How To Fix “Device eth0 does not seem to be present, delaying initialization” Error!!

Try to start Eth0 device:
# ifup eth0
Device eth0 does not seem to be present, delaying initialisation

First step – check the MAC Addresses are set correctly:-
[root@centOS network-scripts]#    cat ifcfg-eth0 | grep HWA

HWADDR=”08:00:27:FC:73:2A”

[root@centOS network-scripts]#    cat ifcfg-eth1 | grep HWA

HWADDR=”08:00:27:09:E1:75?

[root@centOS network-scripts]#    cat ifcfg-eth2 | grep HWA

HWADDR=”08:00:27:31:2D:D6?

[root@centOS network-scripts]# ip -o link

1: lo: mtu 16436 qdisc noqueue state   UNKNOWN \    link/loopback   00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0: mtu 1500 qdisc   pfifo_fast state UP qlen 1000\      link/ether 08:00:27:fc:73:2a brd ff:ff:ff:ff:ff:ff

3: eth8: mtu 1500 qdisc   pfifo_fast state UP qlen 1000\      link/ether 08:00:27:09:e1:75 brd ff:ff:ff:ff:ff:ff

4: eth2: mtu 1500 qdisc   pfifo_fast state UP qlen 1000\      link/ether 08:00:27:31:2d:d6 brd ff:ff:ff:ff:ff:ff

Theres no eth1 but there is an eth8. Lets check what is there in dmesg:
root@centOS network-scripts]# dmesg | grep eth1

e1000 0000:00:08.0: eth1: (PCI:33MHz:32-bit) 08:00:27:09:e1:75

e1000 0000:00:08.0: eth1: Intel(R) PRO/1000 Network Connection

udev: renamed network interface eth1 to eth8.

So udev decided to rename my interface, Why is this? A little look at /etc/udev/rules.d/70-persistent-net.rules revealed the answer:-
[root@centOS ~]# cat /etc/udev/rules.d/70-persistent-net.rules |   grep “08:00:27:09:e1:75?

SUBSYSTEM==”net”, ACTION==”add”,   DRIVERS==”?*”, ATTR{address}==”08:00:27:09:e1:75?,   ATTR{dev_id}==”0x0?, ATTR{type}==”1?,   KERNEL==”eth*”, NAME=”eth1?

SUBSYSTEM==”net”, ACTION==”add”,   DRIVERS==”?*”, ATTR{address}==”08:00:27:09:e1:75?,   ATTR{dev_id}==”0x0?, ATTR{type}==”1?,   KERNEL==”eth*”, NAME=”eth8?

Looks like there are mulitple entries for the same MAC address. Removed the incorrect entry and restart the interface using below command.
# ifup eth0
Thats it..

Cron

The software utility Cron is a time-based job scheduler in Unix-like computer operating systems. People who set up and maintain software environments use cron to schedule jobs (commands or shell scripts) to run periodically at fixed times, dates, or intervals. It typically automates system maintenance or administration—though its general-purpose nature makes it useful for things like connecting to the Internet and downloading email at regular intervals.

Cron is driven by a crontab (cron table) file, a configuration file that specifies shell commands to run periodically on a given schedule. The crontab files are stored where the lists of jobs and other instructions to the cron daemon are kept. Users can have their own individual crontab files and often there is a system wide crontab file (usually in /etc or a subdirectory of /etc) that only system administrators can edit.

For Scheduling Repetitive Jobs: crontab

You can schedule routine system administration tasks to execute daily, weekly, or monthly by using the crontab command.
Daily crontab system administration tasks might include the following:

• Removing files more than a few days old from temporary directories
• Executing accounting summary commands
• Taking snapshots of the system by using the df and ps commands
• Performing daily security monitoring
• Running system backups

Weekly crontab system administration tasks might include the following:

• Rebuilding the catman database for use by the man -k command
• Running the fsck -n command to list any disk problems

Monthly crontab system administration tasks might include the following:

• Listing files not used during a specific month
• Producing monthly accounting reports

Controlling Access to the crontab Command

You can control access to the crontab command by using two files in the /etc/cron.d directory: cron.deny and cron.allow. These files permit only specified users to perform crontab command tasks such as creating, editing, displaying, or removing their own crontab files.
The cron.deny and cron.allow files consist of a list of user names, one user name per line.
These access control files work together as follows:

• If cron.allow exists, only the users who are listed in this file can create, edit, display, or remove crontab files.
• If cron.allow does not exist, all users can submit crontab files, except for users who are listed in cron.deny.
• If neither cron.allow nor cron.deny exists, superuser privileges are required to run the crontab command.

Superuser privileges are required to edit or create the cron.deny and cron.allow files.

Note:

• Check for the job errors in /var/spool/mail/username file.
• Escape percent (%) sign with backslash in command line while scheduling a job.

QMole – The World of Linux on an IPad Near You

Have you noticed that many stock applications which are free on Linux require payment on the IPhoneTM  and IPad TM? Don’t want to re-implement Linux software on IOSTM ?  QMole is the answer. QMole is a new desktop system allowing the free operation of software ported from the world of Linux on the IPad. The technology enables retaining a touch screen operation of stock GTK Linux applications without requiring their redesign or reimplementation. Unlike remote desktop solutions that require a network connection, QMole requires none. All “Linux” applications execute locally on iPad TM, just like native IOS TM applications.

http://youtu.be/ofk0M1LjdtU

Print paragraph between semicolons containing a pattern

Below two awk snippets can do this for you:

• awk 'function dothis(){ if(v && s==2){ print p} if(s==2)s=v=p="" }{ dothis(); s += /;/ ; v += /XXXX/; s ? p = length(p) ? p ORS $0 : $0 : "" }END{dothis()}' file

   

• awk '($0 ~ /;/){set++;} set{v=v?v ORS $0:$0} (v ~ /XXXX/){value=1} {if(set>1 && value==1){print v;v="";value="";set=""}}'  filename 

Configure/change IP in CentOS/Linux

From command line run following command:

ifconfig
e.g. ifconfig 10.10.10.10 eth0

This will immediately assign the IP to the system. But this will be removed after reboot.

To make it permanent, edit below file:

/etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
BOOTPROTO=static
DHCPCLASS=yes
IPADDR=10.10.10.10
NETMASK=255.255.255.0
ONBOOT=yes

.....

Now define gateway in file:

/etc/sysconfig/network

NETWORKING=yes
HOSTNAME=manutablet
GATEWAY=192.168.1.254

 ..

Once these are defined, restart network services:
service network restart

Note: Do not stop/start service. Only restart it otherwise after stop you will lose your connection and you will need to go to the server and start the services from console.

Yummy yum for CentOSians

Lets say someone wants to install a package, or update a package or just wanted to check if there is any update of a package. The command "yum" would be pretty helpful for them. The basic syntax for yum would be like below:

yum install
Used to install the latest version of a package or group of packages.

yum update
Used to update the specified packages to the latest available version.

yum update stuck due to broken package: (Lets say due to firefox and java packages)

                                  vi /etc/yum.conf --> In main section add below line:
                                  exclude=firefox* java*

Then run the update again like below:
                                  yum update --skip-broken

yum check-update
This command allows you to determine whether any updates are available for your installed packages. yum returns a list of all package updates from all repositories if any are available.

yum localinstall
Used when using yum to install a package located locally in the machine

yum provides
Used to determine which packages provide a specific file or feature.

yum search
This command is used to find any packages containing the specified keyword in the description, summary, packager and package name fields of RPMs in all repositories.

To search for a specific package by name, use the list function. To search for the package tsclient, use the command: (provide root's password when prompts)

su -c 'yum list tsclient'

What about uninstalling a package? Yes yum will again help you in this regard.

yum remove
Used to remove specified packages, along with any other packages dependent on the packages being removed.

By default, yum is configured through /etc/yum.conf.

cat /etc/yum.conf

[main]
cachedir=/var/cache/yum
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
pkgpolicy=newest
distroverpkg=redhat-release
tolerant=1
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
metadata_expire=1800

[myrepo]
name=RHEL 5 $releasever - $basearch
baseurl=http://local/path/to/yum/repository/
enabled=1
A typical /etc/yum.conf file is made up of two types of sections: a [main] section, and a repository section. There can only be one [main] section, but you can specify multiple repositories in a single /etc/yum.conf.

Extract rpm/jar/tar/iso in Windows 7

Windows 7-Zip from All Programs Menu

How about extracting .rpm/iso/tar/jar files on windows 7 machine?

Once of the best utilities i have come across in recent years in 7-Zip. I was particularly interested in jar, tar, iso and rpm files extraction. Other than these four  extensions, 7-Zip supports a number of other compression and non-compression archive formats (both for packing and unpacking) including 7z, ZIP, GZip, bzip2, xz and WIM. The utility also supports unpacking APM, ARJ, CHM, cpio, DEB, FLV, JAR, LHA/LZH, LZMA, MSLZ, onepkg, RAR, RPM, smzip, SWF, XAR and Z archives and CramFS, DMG, FAT, HFS, ISO, MBR, NTFS, SquashFS, UDF and VHD disk images.

7-Zip can open some MSI files, allowing access to the meta-files within along with the main contents. Some Microsoft CAB (LZX compression) and NSIS (LZMA) installer formats can be opened. Similarly, some Microsoft executable programs (.EXEs) which are self-extracting archives or otherwise contain archived content (e.g., some setup files) may be opened as archives.

The utility can be downloaded from here:
http://www.7-zip.org/download.html

fuser and pfiles: lsof equivalent in Solaris

Lets talk about lsof. It displays information about files which are opened by running(active) processes. An open file may be a regular file like text file or any script, directory, NFS file, block special file, character special file, shared library, regular pipe, named pipe, symbolic link, socket, anything. In Unix, we consider everything is a file, so you can think of how incredibly smart utility lsof is. Generally it is supplied with Linux.

But the question is, what is its Solaris equivalent? 

After some hit and misses i have managed to run an equivalent command. fuser. The fuser syntax that worked for me is :

sudo fuser -vm / 2>&1 | awk '$3 ~ /f|F/' | less

Another one is pfiles. It is not necessarily supplied with the default package.

pfiles /proc/*

Run fsck on a filesystem to repair it..

fsck -y -o b=32 /filesystem

This will repair the corrupted file system to its best and restore the data in /filesystem/lost+found directory. Data in this directory will have "#inode name" nomenclature. From there the data can be copied back to the filesystem. Ofcourse the techie who is repairing the FS must have prior knowledge about the directory structure in filesystem otherwise he would not be able to decide where to copy which file from lost+found directory. 

Note: This is system administrator's work, so please do not run mount/umount/fsck sort of commands with minimal knowledge. The fsck command can have other parameters as well. So depending on the need of the work, one can decide the exact fsck command. Above fsck command is most generic one i came across and applied while **working on real time issues**. Also please keep in mind that the filesystem has to be unmounted first (umount /filesystem) then run fsck and then mount (mount /filesystem /mountpoint) the filesystem back if properly repaired. Then only you will be able to see the files for restore in /lost+found directory.

Mount ISO image in Linux

Getting a lot of emails regarding mounting ISOs in linux. Please use the following steps in order to mount the image to FS.

Procedure to mount ISO images in Linux.

1) Login as a root user:
$ su -

2) Create a mount point:
# mkdir -p /mnt/disk

3) Mount iso file called mountthisdisk.iso:
# mount -o loop mountthisdisk.iso /mnt/disk

4) Verify it:
# cd /mnt/disk
# ls -l

In case you have not noticed "loop" in the mount command, let me tell you that A loop device is a pseudo-device that makes a file accessible as a block device. Loop devices are often used for CD ISO images and floppy disc images. Mounting a file containing a filesystem via such a loop mount makes the files within that filesystem accessible. They appear in the mount point directory using above commands.

Find files which are older than 15 minutes in Linux and Solaris..

The below example is for moving (mv) files, same can be changes easily for rm, ls, cp etc. One has to make change in xargs paramaeter.

In LINUX:
=========

find /path_to_directory/ -name "*.rpt" -type f -mmin +15 | xargs -I {} mv {} /target_directory_path/


In Solaris:
===========

#! /usr/bin/sh

#### Create a reference file minute_test
touch /path_to_directory/minute_test

#### Make minute_test 15 minutes old
sleep 900

#### Move all .rpt files to some other directory which are older than reference file minute_test
find /path_to_directory/ -name "*.rpt" -type f -newer minute_test | xargs -I {} mv {} /target_directory_path/

#### Remove reference file
rm -f /path_to_directory/minute_test

Count the number of occurances of a pattern in a file using awk

awk '{ for (i=1;i<=NF;i++) if ( $i == "word" ) count++ } END{print count}' filename

This will print the count of the occurance of  a pattern "word" in a file.

How to compare the values of a column in awk in a same file and consecutive lines..

If one would like to compare the values of 2nd column of consecutive lines of same file in such a way so that if the difference between first value and second value is more than 100 it should print complete line else ignore line.

Input File:

=======

ABC 2500

ABCD 123

XYZ 122

WXYZ 2565

Desired Output:

==========

ABC 2500 (i.e. difference between 2500 and 123 is greater than 100 here)

XYZ 122 (i.e. difference between 122 and 2565 is greater than 100 here)

Command:

=======

awk 'NR % 2 != 0 {a=$1; b=$2} NR % 2 == 0 {if (b - $2 > 100 || $2 - b > 100){print a,b}}' inputfile

Count the number of files in each directory recursively

find ./ -type d -exec sh -c "echo -n {} ' ' ; ls -l {} | wc -l" \;

Print 3 lines before and 5 lines after pattern match abcd in a file a.txt

In Linux:

awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=3 a=5 s="abcd" a.txt

In SOLARIS:

nawk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=3 a=5 s="abcd" a.txt

ftp in UNIX

ftp -n 10.200.120.12 << EOF
quote user username
quote pass password
cd /path/to/directory/
prompt off
put dump.txt
exit
EOF

Adding 30 days to current date using PERL inside SHELL

#! /usr/bin/ksh

dt1=`perl -l -e '$date=(localtime(time + (30 * 24 * 60 *60))); print $date;'`

dt=`echo $dt1|awk '{if($3<10) print $1" "$2" 0"$3" "$4" "$5;else print $0}'`

echo $dt|awk '{print $5"-"$2"-"$3"23:59:59"}'|sed -e 's/Jan/01/g'\;'s/Feb/02/g'\;'s/Mar/03/g'\;'s/Apr/04/g'\;'s/May/05/g'\;'s/Jun/06/g'\;'s/Jul/07/g'\;'s/Aug/08/g'\;'s/Sep/09/g'\;'s/Oct/10/g'\;'s/Nov/11/g'\;'s/Dec/12/g'

Print 3 lines before pattern match using sed and awk ..

To reverse a file using awk and print next three lines after pattern match (Including pattern):

awk '{ a[i++] = $0 } END { for (j=i-1; j>=0;) print a[j--] }' abcd.txt |sed -n '/MANU/{p;n;p;n;p;n;p;}' >abcd_new.txt

(This can be used when you want to print 3 lines before pattern match, including pattern. Logic: reverse a file and print 3 lines after pattern match.)

How to reverse a file using awk:

awk '{ a[i++] = $0 } END { for (j=i-1; j>=0;) print a[j--] }' abcd.txt

How to reverse a file using sed:

sed -n '1!G;h;$p' abcd.txt

Print one line before pattern match using sed in UNIX.

sed -n -e '/manu/{x;p;x;p;}' -e h test.txt