BigData Technology Landscape

BigData Technology Landscape

Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.

Big data analytics is the process of examining big data to uncover hidden patterns, unknown correlations and other useful information that can be used to make better decisions.

BigData Technology Landscape

9 ways to tell if you’re really a DevOps organization

9 ways to tell if you’re really a DevOps organization
Many people believe that their organizations use a DevOps approach, but the “best practices” they’ve adopted aren’t consistent with a DevOps methodology. You can call what you do at your organization “DevOps,” but if you don’t follow the basic software methodology principles, it’s not DevOps. Period.

Here’s a set of objective criteria you can use to determine whether what you’re doing is truly DevOps. You’re really doing DevOps if…

1. You have frequent, rapid release cycles.
2. Deployment to production is fully automated, and you can automatically roll back.
3. You do continuous integration with automated testing at every check-in.
4. You have far more resources devoted to automated testing than manual testing.
5. Your developers, testers, and operations engineers work together.
6. Your developers, testers, and operations engineers have common, business-oriented goals that cover the whole value chain.
7. You’re paying back your technical debt.
8. Management gives the team the authority to make the changes it needs to make—and the team makes those changes.
9. You have feedback mechanisms in place to ensure continuous assessment—and act on that feedback to ensure continuous improvement.

Step by Step Procedure To Setup Cassandra Cluster On Any Linux Distribution

Step by Step procedure to set-up Cassandra Cluster On Any Linux Distribution

1. What is Cassandra : Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure.

2. Cassandra Cluster: A cluster is arranged as a ring of nodes. Clients send read/write requests to any node in the ring; that node takes on the role of coordinator node, and forwards the request to the node responsible for servicing it. A partitioner decides which nodes store which rows. The recommended partitioners assign rows to nodes based on a hash value of the row key. Nodes are assigned tokens that evenly divide the full range of possible hash values.

3. Prerequisites :
Setup iptables, allow below Cassandra ports.
For Centos:
iptables -A INPUT -p tcp -m tcp –dport 9160 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7000 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7001 -j ACCEPT
service iptables restart

For Ubuntu:
iptables -A INPUT -p tcp -m tcp –dport 9160 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7000 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7001 -j ACCEPT
sudo sh -c “iptables-save > /etc/iptables.rules”
sudo iptables-apply /etc/iptables.rules
Applying new iptables rules from ‘/etc/iptables.rules’… done.
Can you establish NEW connections to the machine? (y/N) y
… then my job is done. See you next time.

4. Installing Cassandra and Configuring: Login to Remote machine, where you want to install Cassandra.
Below are the steps to install and configure Cassandra Cluster.

Get Cassandra tar-ball
wget “http://downloads.datastax.com/community/dsc-cassandra-2.1.7-bin.tar.gz” -P /usr/share/
Setup Cassandra environment under /usr/share
tar -xvzf /usr/share/dsc-cassandra-2.1.7-bin.tar.gz -C /usr/share/

Delete tar file
rm -rf /usr/share/dsc-cassandra-2.1.7-bin.tar.gz

Cassendra configuration change cluster_name : change clustername with your Cluster
sed -i ‘s/Test Cluster/{{clustername}}/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change listen_address : hostname=serverip
sed -i ‘s/listen_address\:\ localhost/listen_address\:\ {{hostname}}/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change rpc_address
sed -i ‘s/rpc_address\:\ localhost/rpc_address\:\ 0.0.0.0/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change endpoint_snitch
sed -i ‘s/endpoint_snitch\:\ SimpleSnitch/endpoint_snitch\:\ GossipingPropertyFileSnitch/g’ /usr/share/dsc-
cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change broadcast_rpc_address hostname=serverip
sed -i ‘s/^# broadcast_rpc_address\:\ 1.2.3.4/broadcast_rpc_address\:\ {{hostname}}/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change seeds hostname=serverip seeds=seedsip
sed -i ‘s/- seeds\:\ “127.0.0.1”/- seeds\:\ “{{seeds}},{{hostname}}”/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change data_file_directories
sed -i ‘s/# data_file_directories/data_file_directories/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change /var/lib/cassandra/data
sed -i” ‘107 s/^#//’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change commitlog_directory
sed -i ‘s/# commitlog_directory/commitlog_directory/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change saved_caches_directory
sed -i ‘s/# saved_caches_directory/saved_caches_directory/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change thrift_framed_transport_size_in_mb
sed -i ‘s/thrift_framed_transport_size_in_mb\:\ 15/thrift_framed_transport_size_in_mb\:\ 100/g’ /usr/share/dsc-
cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change phi_convict_threshold
sed -i ‘s/# phi_convict_threshold\:\ 8/phi_convict_threshold\:\ 10/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Start Cassandra service
nohup sh /usr/share/dsc-cassandra-2.1.7/bin/cassandra

5. Command to check Cassandra cluster:
/usr/share/dsc-cassandra-2.1.7/bin/nodetool status
Address Status State Load Owns Token
113427455640312821154458202477256070484
10.1.1.7 Up Normal 459.27 MB 33.33% 0
10.1.1.8 Up Normal 382.53 MB 33.33% 56713727820156410577229101238628035242
10.1.1.6 Up Normal 511.34 MB 33.33% 113427455640312821154458202477256070484

DevOps Tools

Here are some of the DevOps tools.

  1. Operating Systems
    1. Linux (RHELCentOSUbuntuDebian)
    2. Unix (SolarisAIXHP/UX, etc.)
    3. Windows
    4. Mac OS X
  2. Infrastructure as a Service
    1. Amazon Web Services
    2. Rackspace
    3. Cloud Foundry
    4. Azure
    5. OpenStack
  3. Virtualization Platforms
    1. VMware
    2. KVM
    3. Xen
    4. VirtualBox
    5. Vagrant
  4. Containerization Tools
    1. LXC
    2. Solaris Containers
    3. Docker
  5. Linux OS Installation
    1. Kickstart
    2. Cobbler
    3. Fai
  6. Configuration Management
    1. Puppet / MCollective
    2. Chef
    3. Ansible
    4. CFEngine
    5. SaltStack
    6. RANCID
    7. Ubuntu Juju
  7. Test and Build Systems
    1. Jenkins
    2. Maven
    3. Ant
    4. Gradle
  8. Application Deployment
    1. Capistrano
  9. Application Servers
    1. JBoss
    2. Tomcat
    3. Jetty
    4. Glassfish
    5. Websphere
    6. Weblogic
  10. Web Servers
    1. nginx
    2. Apache
    3. IIS
  11. Queues, Caches, etc.
    1. ActiveMQ
    2. RabbitMQ
    3. memcache
    4. varnish
    5. squid
  12. Databases
    1. Percona Server
    2. MySQL
    3. PostgreSQL
    4. OpenLDAP
    5. MongoDB
    6. Cassandra
    7. Redis
    8. Oracle
    9. MS SQL
  13. Monitoring, Alerting, and Trending
    1. New Relic
    2. Nagios
    3. Icinga
    4. Graphite
    5. Ganglia
    6. Cacti
    7. PagerDuty
    8. Sensu
    9. Zabbix
    10. Solarwinds
    11. Application Manager
  14. Logging
    1. PaperTrail
    2. Logstash
    3. Loggly
    4. Splunk
    5. SumoLogic
  15. Process Supervisors
    1. Monit
    2. runit
    3. Supervisor
    4. god
    5. Blue Pill
    6. Upstart
    7. systemd
  16. Security
    1. Snorby Threat Stack
    2. Tripwire
    3. Snort
  17. Miscellaneous Tools
    1. Multihost SSH Wrapper
    2. Code Climate
    3. iPerf
    4. lldpd

got this useful information from http://newrelic.com/devops/toolset

Step by Step procedure to install and Configure ANSIBLE on Ubuntu

Step by step procedure to install and configure ansible on ubuntu

Before installing ANSIBLE, let us understand,

What is ANSIBLE ?

1.Ansible Introduction :

Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero
downtime rolling updates.
Ansible’s goals are foremost those of simplicity and maximum ease of use. It also has a strong focus on security and reliability, featuring a minimum of moving parts, usage of OpenSSH for transport (with an accelerated socket mode and pull modes as alternatives),
and a language that is designed around auditability by humans – even those not familiar with the program.

While there are many popular configuration management systems available for Linux systems, such as Chef and Puppet, these are often more complex than many people want or need.Ansible is a great alternative to these options because it has a much smaller overhead to get started.

for more information visit, Ansible docs.
http://docs.ansible.com/

Assumptions
● You have a ubuntu 12.04 and you are logged-in as root.

2.How Does Ansible Work :
Ansible works by configuring client machines from an computer with Ansible components
installed and configured.
It communicates over normal SSH channels in order to retrieve information from remote
machines, issue commands, and copy files. Because of this, an Ansible system does not require any additional software to be installed on the client computers.
This is one way that Ansible simplifies the administration of servers.
Configuration files are mainly written in the YAML data  format due to its
expressive nature and its similarity to popular markup languages.Ansible can interact with clients through either command line tools or through its configuration scripts called Playbooks.

3. Install Ansible on an Ubuntu 12.04 :
To begin exploring Ansible , we need to install the Ansible package on at least one machine. I have used ubuntu 12.04

git clone git@github.com:ansible/ansible.git
cd ansible
source ./hacking/env-setup
sudo pip install paramiko PyYAML jinja2 --upgrade

We now have all of the software required to administer our servers through Ansible.
Set Up SSH Keys , Ansible primarily communicates with client computers through SSH.While it certainly has the ability to handle password-based SSH authentication, SSH keys help keep things simple.

4. Creating a SSH Pair :
@Server:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/a/.ssh/id_rsa):
Created directory ‘/home/a/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/a/.ssh/id_rsa.
Your public key has been saved in /home/a/.ssh/id_rsa.pub.
The key fingerprint is:
3e:4f:05:79:3a:9f:96:7c:3b:ad:e9:58:37:bc:37:e4 a@A
Now use ssh to create a directory ~/.ssh as user b on Client. (The directory may already exist,
which is fine):
@Server:~> ssh root@123.123.123.123 mkdir -p .ssh
root@Client’s password:
Finally append a’s new public key to root@123.123.123.123:.ssh/authorized_keys and enter
client’s password one last time:
root@Server:~> cat .ssh/id_rsa.pub | ssh root@123.123.123.123 ‘cat >> .ssh/authorized_keys’
root@Client’s password:
From now on you can log into Client as root from A as a without password:
@Server:~> ssh root@123.123.123.123

5. Configuring Ansible Host:
Ansible keeps record of all of the servers that it know through a “hosts” file. We need to configure this file first before we can begin to communicate with our other computers.
Open the file and add,

vim /etc/ansible/hosts
[hosts]
devops ansible_ssh_host=123.123.123.123

Where hosts is list of nodes
devops = host name of one of the client.
123.123.123.123 is hosts ip.
To do this, we will create a directory in the Ansible configuration structure called group_vars.
Within this folder, we can create YAML-formatted files for each group we want to configure:

sudo mkdir /etc/ansible/group_vars
sudo nano /etc/ansible/group_vars/hosts

We can put our configuration in here. YAML files start with “—“, so make sure you don’t forget that part.

ansible_ssh_user: root
Save and close this file when you are finished.

6. Ansible test Examples:

Using Simple Ansible Commands
Now that we have our hosts set up and enough configuration details to allow us to
successfully connect to our hosts, we can try out our very first command.
Ping all of the servers you configured by typing:
root@server:/etc/ansible# ansible -m ping hosts
devops | success >> {
“changed”: false,
“ping”: “pong”
}
This is a basic test to make sure that Ansible has a connection to all of its hosts.

ansible -m shell -a ‘ifconfig’ devops

devops | success | rc=0 >>
eth0 Link encap:Ethernet HWaddr 00:0c:29:a2:42:c4
inet addr:123.123.123.123 Bcast:10.27.111.255 Mask:255.255.255.0

ansible -m shell -a ‘free -m’ devops
devops | success | rc=0 >>
total used free shared buffers cached
Mem: 487 440 47 0 108 173
-/+ buffers/cache: 158 329
Swap: 507 66 441

Lets install sendmail on client from Ansible server, to do that create playbook.yml file in
/etc/ansible 100 Jul 5 02:48 playbook.yml

root@server:/etc/ansible# cat playbook.yml

– hosts: devops
tasks:
– name: 1. install sendmail
apt: name=sendmail state=present

Now execute below command to install on remote server.
ansible-playbook –inventory-file=hosts playbook.yml –sudo –verbose
Below is the result,

ansible 1

Before executing above command, sendmail was not installed. ansible 2

Later after execution, sendmail installed on the remote host.

Advantages of Automation,

,Automation-Advantages

Hence we have Installed and Configured ANSIBLE, Enjoy !!!

shell script to replace a string in a file

shell script to replace a string in a file

Some times during automating any task, we may require to replace a string in a file.
Example:- I want to change “Linux” string with “Waseem” in a file called string.txt

cat replacestring.sh
#!/bin/bash
# Shell Script to replace string in a file
/bin/sed -i “s/Linux/Waseem/g” /PATH/TO/FILE/string.txt
exit 0;

Before:-
—————–
cat string.txt
Linux

Run the script:-
——————-
sh replacestring.sh

After:-
——————
cat string.txt
Waseem

Another way we can replace string by passing parameter,so that the script will be used dynamically.
cat replacestring.sh
#!/bin/bash
# Shell Script to replace string in a file
/bin/sed -i “s/$1/$2/g” /PATH/TO/FILE/string.txt
exit 0;

Run the script:-
—————–
sh replacestring.sh Linux Waseem

Time Value of Money

The concept of Time Value of Money:
“A rupee is now worth full than a rupee to be received after a year” why ?

Do you prefer a 1000 rupees today or a 1000 rupees one year from now? why?

– Consumption forgone has value
– Investment lost has opportunity cost
– Inflation may increase and purchasing power decrease

Now,
Do you prefer a 1000 rupees today or 1100 rupees one year from now? Why?

You will ask yourself one question:
– Do I have any thing better to do with that 1000 rupees than lending it for 100 rupees
extra?

– What if I take 1000 rupees now and invest it, would I make more or less than
1100 rupees in one year?

Note:
Two elements are important in valuation of cash flows:
– What interest rate (opportunity rate, discount rate, required rate of
return) do you want to evaluate the cash flow based on?
– At what time do these the cash flows occur and at what time do you need
to evaluate them?

Example:
What return is needed to double money?
The present value formula can be rearranged to determine what rate of return is
needed to accumulate a given amount from an investment. For example, £100 is invested today and £200 return is
expected in five years; what rate of return (interest rate) does this represent?

The present value formula restated in terms of the interest rate is:

tme value of money

In addition to Time value of Money, let us also discuss, what is Opportunity cost?

Definition:
Opportunity cost is the cost of a foregone alternative. If you chose one alternative over another, then the cost of choosing that alternative is an opportunity cost. Opportunity cost is the benefits you lose by choosing one alternative over another one. The opportunity cost of choosing one investment over another one.

Example:
There is an opportunity cost over choosing an investment in bonds over an investment in stocks.

Shell Script : Auto Enable/Disable Specific Cronjob

Shell Script : Auto Enable/Disable Specific Cronjob

What is Cronjob ?
A cron job is a scheduled task that is executed by the system at a specified time/date.

Sometimes we want to perform some task during specific time(Ex: During production Hours), and once the production time completes, we will be disabling cronjob. this requires manual work.The above manual work, can be automated with this script.

Create two scripts,
1.Auto_enable_cron.sh
2. Auto_disable_cron.sh

Below are both the scripts with cronjob details,

COMMANDS used:

crontab -l | sed ‘/^#.*GIVE-SCRIPT-NAME-HERE.sh/s/^#//’ | crontab –
crontab -l | sed ‘/^[^#].*GIVE-SCRIPT-NAME-HERE.sh/s/^/#/’ | crontab –

cronandscript

cronoutput

In the above image we can see that after running Auto_enable_cron.sh
The script in the cron GIVE-SCRIPT-NAME-HERE.sh is uncommented and after running
Auto-disable_cron.sh the script GIVE-SCRIPT-NAME-HERE.sh is commented again.

Now add scripts to desired time in cronjob.
9 * * * * /home/cronjob/Auto_enable_cron.sh (This will enable GIVE-SCRIPT-NAME-HERE.sh @9)
23 * * * * /home/cronjob/Auto_disable_cron.sh (This will disable GIVE-SCRIPT-NAME-HERE.sh @23)

Enjoy, just now you have automated your task!!!!.

Product life cycle

Product life cycle is a business analysis that attempts to identify a set of common stages in the life of commercial products. In other words the ‘Product Life cycle‘ PLC is used to map the lifespan of the product such as the stages through which a product goes during its lifespan.

productlifecycle