Basic Meaning of Big Data

From last many days some students and my juniors are asking about basic of Big Data and how it is related to Cloud Computing. So I thought to write an article explaining meaning Big Data.

As the name suggest Big Data is related with huge amount to data which can not be processed by using simple methods and tools. For example, modern high-energy physics experiments, such as DZero1, typically generate more than one TeraByte of data per day. The famous social network Website, Facebook, serves 570 billion page views per month, stores 3 billion new photos every month, and manages 25 billion pieces of content. Main Question is how to process this much of data in less time? Data collected from these sources is very loosely linked to each other so making decisions from this data is very complex and time consuming. Our today's conventional databases cannot process data if they don't know exact relation between terms. Now a days organizations collect data form many different sources and methods. For example, an laptop company can collect data about a product from social networking sites such as Facebook, twitter etc. from many laptop related blogging websites and even from laptop selling online sites. Data collected from this much different sources can’t be process my conventional databases to make any proper decisions. This data is too big, moves too fast, or doesn’t fit the structure of conventional databases. To gain value from this data, we must follow a new approach. This new approach is known as Big Data.

Seeing the need of Big Data on March 29, 2012 American Government announced “Big Data Research and Development Initiative” and Big Data became the national policy for first time. Recently, the definition of big data as also given by the Gartner: “Big Data are high-volume, high-velocity, and/or high-variety information assets that require new forms of processing to enable
enhanced decision making, insight discovery and process optimization”( I will explain three V’s of big data in coming posts in detail).

Basically technique used for getting useful information from very large unstructured data sets is known as Big Data. This data can be of any type. I hope you get very basic meaning of Big Data. Stay tuned i will post for detail about Big data architecture, 3 V’s and all in subsequent posts.

Rajinder Sandhu

Hadoop on Multi Node

This tutorial provides step by step way to install multinode Hadoop on Ubuntu 10.04.

1. Install Single node Hadoop on all Nodes before starting to follow this tutorial.

2. If you follow my single node tutorial and install Ubuntu 10.04 on VMware Workstation. You can clone your single node machine using method as shown in Figure below.



3. Name the cloned virtual machine as slave. This can be change from System>Administration>User and Groups.

1. Networking of Master and Slave

1.1 All nodes should be accessable from each other on the network. 

1.2 Add IP address of both master and slave machine in all the machines using following command.

• ifconfig # on all machines to know their IP.
• sudo nano /etc/hosts

Add following lines in the file.

• 192.168.216.135 master
• 192.168.216.136 slave

2. SSH Access

SSH access should be enabled form master to slave so that jobs can be transfered from master to slave and vice versa.

2.1 Add RSA key of slave to authorized users using following command.

• su - hduser
• ssh-copy-id -i $HOME/.ssh/id_rsa.pub hduser@slave

Now check the working of SSH by following commands.

• ssh master
• ssh slave

3. Configuration of Hadoop

3.1 Add IP and name of masters in /conf/master file of master node only using following command.

• su - hduser
• cd /home/hadoop/conf
• nano masters

Add name of masters in this case:

• master

and then

• nano slaves

Add name of slaves in this case:

• master
• slave

3.2 Change different parameter files of hadoop files on all machines as follows, as changed in single node installation.

In conf/core-site.xml:

<property> <name>fs.default.name</name> <value>hdfs://master:54310</value> Only line which require change. <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description> </property>

In conf/mapred-site.xml

<property> <name>mapred.job.tracker</name> <value>master:54311</value> Only Line require the change <description>The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. </description> </property>

In conf/hdfs-site.xml:

set replication value to 2 which was 1 eariler.

3.3 Format the NameNode

• /home/hadoop/bin/hadoop namenode -format

3.4 If namenode donot get formatted delete temp folder of hadoop

• cd /app/hadoop/tmp
• rm -R dfs
• rm -R mapred

4. Starting the Multi node Hadoop

4.1 Run the command bin/start-dfs.sh on the machine you want the (primary) NameNode to run on. This will bring up HDFS with the NameNode running on the machine you ran the previous command on, and DataNodes on the machines listed in the conf/slaves file.

4.2 Run the command bin/start-mapred.sh on the machine you want the JobTracker to run on. This will bring up the MapReduce cluster with the JobTracker running on the machine you ran the previous command on, and TaskTrackers on the machines listed in the conf/slaves file.

Install Hadoop on Single Node

In this tutorial i will give you line by line instruction on how to setup hadoop on single node. First we will install hadoop on single node then move to multiple node in coming tutorials.

Pre-Requirements:

1. Java

* Update the system files using following command.

• sudo add-apt-repository ppa:webupd8team/java
• sudo apt-get update
• sudo apt-get install oracle-java7-installer

* Now check the version of java 

• java -version

2. Python environment.

Setup Python environment using following command.

• sudo apt-get install python-software-properties

3. Create a new user for Hadoop.

Create a new user for hadoop. We will provide all permission to this user so that our hadoop can be secure.

• sudo addgroup hadoop
• sudo adduser --ingroup hadoop hduser

4. Configure SSH

SSH is required for hadoop to communicate between master and slave nodes. Configure it using following commands

• sudo apt-get install openssh-server
• su –  hduser
• ssh-keygen -t rsa -P “”
• cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Check if ssh is working by using following command.

• ssh localhost

5. Disable IPV6

Sometime due to IPV6 configuration master node of hadoop is not able to communicate with slaves. It is better to disable IPV6 configuration using System Control file by following command.
* Open system control file in nano editor.

• sudo nano /etc/sysctl.conf

* Embedd below written lines to end of system control file.

# disable ipv6

net.ipv6.conf.all.disable_ipv6 = 1

net.ipv6.conf.default.disable_ipv6 = 1

net.ipv6.conf.lo.disable_ipv6 = 1

* Restart the system for settings to take effect. Check if IPv6 is disabled by following command. Output should be 1 when Ipv6 is disabled.

• cat /proc/sys/net/ipv6/conf/all/disable_ipv6

All pre-requirements are complete. 

We start with installation of hadoop on single node cluster.

1. Download hadoop from here.

* Extract the tar file. Rename it to hadoop and save on Desktop.

* Move it to /home folder so that it canbe used by slave nodes also.

• cd Desktop
• sudo mv hadoop /home

* Change owner of this folder to hduser by following command.

• cd /home
• sudo chown -R hduser:hadoop /home/hadoop

* Open .bashrc file by using following command to set java environment variables.

• su - hduser
• nano $HOME/.bashrc

*Add the following lines to the end of the file.
# Set Hadoop-related environment variables

export HADOOP_HOME=/home/hadoop
# Set JAVA_HOME (we will also configure JAVA_HOME directly for Hadoop later on)
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
# Some convenient aliases and functions for running Hadoop-related commands
unalias fs &> /dev/null
alias fs="hadoop fs"
unalias hls &> /dev/null
alias hls="fs -ls"
# If you have LZO compression enabled in your Hadoop cluster and
# compress job outputs with LZOP (not covered in this tutorial):
# Conveniently inspect an LZOP compressed file from the command
# line; run via:
#
# $ lzohead /hdfs/path/to/lzop/compressed/file.lzo
#
# Requires installed 'lzop' command.
#
lzohead () {
    hadoop fs -cat $1 | lzop -dc | head -1000 | less
}
# Add Hadoop bin/ directory to PATH
export PATH=$PATH:$HADOOP_HOME/bin

2. Configure Hadoop script file

* The only required environment variable we have to configure for Hadoop in this tutorial is Java. Follow following commands.

• su - hduser
• cd /home/hadoop/conf
• nano hadoop-env.sh

* and set java home as following:

• # The java implementation to use.  Required.
• export JAVA_HOME=/usr/lib/jvm/java-7–oracle

3. Create a temporary directory for hadoop.

A temp folder is created so that hadoop can store its temporary files in this folder.

• exit
• sudo mkdir -p /app/hadoop/tmp
• sudo chown hduser:hadoop /app/hadoop/tmp

4. Set Configuration files

All hadoop configuration files are set in this section.Add the following snippets between the <configuration> ... </configuration> tags in the respective configuration XML file.
In file conf/core-site.xml.

• su - hduser
• cd /home/hadoop/conf
• nano core-site.xml

Add following between configuration tags:

<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

In file conf/mapred-site.xml:

<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>

In file conf/hdfs-site.xml:

<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>

9. Formatting the HDFS filesystem via the NameNode

• /home/hadoop/bin/hadoop namenode -format

Output will look like:

10. Start the single node cluster

• /home/hadoop/bin/start-all.sh 

Output will look like:

Hadoop is running on this machine check its working using following command.

• cd /home/hadoop
• jps

Above screen shows port where hadoop API's are running.

Google App Engine Tutorial- Part 2

Part 2 of video seriesfor Google App Engine…

Click for Part 1

Virtualization- Basic Meaning

Hi everyone. In my previous post i talk about three basic requirement for Cloud Computing to take hold in the market. First among them was Virtualization. From long time I want to talk about Virtualization in detail but didn't get time. Today I am free, so lets dig deeper in understanding basic meaning of virtualization and How it came to picture?

Virtualization is not the technology of 21^st  century. It was used by IBM for its Mainframe Computers in 1960's. Mainframe computers has multiple resources to consolidate these different resources and make them act like single resource requires Virtualization. Cloud Computing utilize this idea more comprehensively which makes it first requirement for implementation of Cloud Computing. First look at few definitions of Virtualization.

According to Wikipedia “Virtualization in computing, is the creation of a virtual (rather than actual ) version of something , such as hardware platform, operating system, a storage device or a network device.”

Gartner stated that, “ Virtualization is abstraction of IT resources in a way that masks the physical nature and boundaries of those resources from resource user. An IT resource can be Server, Client, Storage, Networks, Application or Operating System”

What we pull out from these two definitions is Virtualization is conversion of Physical to Logical of everything and anything and logical doesn't know about actual physical resource. Just remember Physical to Logical. One more thing if you simply search on web for definitions on virtualization you may find some definitions saying virtualization is running multiple operating System on single hardware resource. This is not definition of virtualization it is a type of virtualization i.e. Server Virtualization. So, don't confuse yourself when you find these type of definitions. Virtualization can be done of anything i.e. Network Virtualization, Storage Virtualization etc.

These are all formal definitions and explanations. I hardly understand any concept with formal definition. I need examples and detailed definition to grab a concept or technology. If you say we started using virtualization with mainframe computers that may be wrong. Actually we were using this technology in our day to day life in one or another way, for example.

Suppose, I have a big shop. No one want it on rent because it is very large and no one can afford that much rent also. What we do in our day to day life. We create a partition in middle of shop and make it into two shops. Both shops are rented to 2 different shopkeepers. Each shopkeeper thinks he rented the whole shop. Actually floor and ceiling is shared. What owner of big shop did in reality–  He VIRTUALIZED the big shop and created two small virtual shops. Same concept we are using for IT resources. Physical resources (Server, Storage, Network) are shared among different virtual machines but each virtual machine is isolated from each other.

Now to do Server Virtualization we need a software known as Hypervisor. The most crucial piece of any virtual infrastructure is the hypervisor, which is what makes server virtualization possible. A hypervisor creates a virtual host that hosts virtual machines. It is also responsible for creating the virtual hardware that VMs will use. If you look up the term hypervisor, the definition will likely say that a hypervisor is an “abstraction layer.” That’s because it abstracts the traditional server operating system (OS) from the server hardware. Another way of saying this is that the hypervisor decouples the OS from the hardware. Your server OS no longer has to be tied to physical hardware and the newly virtualized server can be hardware-independent and containerized inside a virtual machine. There are two types of Hypervisor as shown in Figure below:

Type 1 hypervisor is installed directly on physical server hardware, thus replacing the existing OS. This is the most efficient design, in that it offers the best performance as well as the most enterprise-level data center features. Examples are VMware vSphere and Microsoft Hyper-V.

Type 2 hypervisor is installed and “hosted” by the existing OS and Virtual Machines are known as Guest OS. This is less efficient but enables you to keep existing applications already installed on the host OS. Examples are VMware Workstation, VMware Fusion and Windows Virtual PC.

Hope you now understand basic meaning of Virtualization. If you gave a deep thought on this concept you can easily see at many places we use this concept in our life.

See you all soon and till then happy Virtualizing.

Google App Engine Video Tutorial Series- Part 1

Setup NFS (Network File System) on Ubuntu 10.04 LTS

NFS allows a system to share directories and files with others over a network. By using NFS, users and programs can access files on remote systems almost as if they were local files.

Some of the most notable benefits that NFS can provide are:

• Local workstations use less disk space because commonly used data can be stored on a single machine and still remain accessible to others over the network.

• There is no need for users to have separate home directories on every network machine. Home directories could be set up on the NFS server and made available throughout the network.

• Storage devices such as floppy disks, CDROM drives, and USB Thumb drives can be used by other machines on the network. This may reduce the number of removable media drives throughout the network.

In this post I will show you how to setup NFS on ubuntu 10.04.

Experimental Setup: For demonstration of nfs. My experimental setup includes software such as- Oracle VirtualBox, Ubuntu 10.04 iso image. I have window 7 as host operating system. I install two virtual ubuntu 10.04 using oracle virtualbox. So now i have two ubuntu machines running on Oracle virtual box. Both machines are accessible to each other on network.

On Server Side:

Start the server ubuntu machine.

1. First of all install nfs kernel. Internet is required for this.

sudo apt-get install nfs-kernel-server

2. Make directory you want to mount. Always create a new directory never mount your operating system base directories.

sudo mkdir –p /export/users

3. Change mode of these directories so that anyone can read this directory. Change mode according to your need. I am changing it to 777 as least secure.

sudo chmod 777 /export
sudo chmod 777 /export/users

4. Now if you want to share any base operating system folder you can bind it with mounted folder.(server is name of my machine)

sudo mount --bind /home/server /export/users

5. Above binding will be removed when you restart your system. To keep this binding permanently open

sudo nano /etc/fstab

add following line

/home/server /export/users none bind 0 0

6. Now open-

sudo nano /etc/default/nfs-kernel-server

change or make

NEED_SVCGSSD = no

7. Now open

sudo nano /etc/default/nfs-common

make or change

NEED_IDMAPD = yes
NEED_GSSD = no

8. make sure value in /etc/idmapd.conf following lines are there.

cat /etc/idmapd.conf

check:

Nobody_user = nobody
Nobody_group = nogroup

9. Most important step add this mounted folder in export file. (192.168.80.136 is my server ip)

Open

sudo nano /etc/exports

add following lines at the end

/export         192.168.80.136/24(rw,fsid=0,insecure,no_subtree_check, async)
/export/users   192.168.80.136/24(rw,nohide,insecure,no_subtree_check,async)

10. Now restart the nfs kernel.

sudo /etc/init.d/nfs-kernel-server restart

On client Side:

1. Install nfs common on client side. It requires Internet Connection.

sudo apt-get install nfs-common

2. Open

sudo nano /etc/default/nfs-common

Set:

NEED_IDMAPD = yes
NEED_GSSD = no

3. Now mount the exported folder.

sudo mount –t nfs4 –o proto=tcp, port=2049 192.168.80.136:/ /mnt

mnt is folder created on client side where all files will be mounted.

That's all your nfs is complete.

Error:

If following error came.

mount.nfs4: No such device

you have to load modprobe module by following command.

sudo modprobe nfs