Category Archives: Technical

9 ways to tell if you’re really a DevOps organization

9 ways to tell if you’re really a DevOps organization
Many people believe that their organizations use a DevOps approach, but the “best practices” they’ve adopted aren’t consistent with a DevOps methodology. You can call what you do at your organization “DevOps,” but if you don’t follow the basic software methodology principles, it’s not DevOps. Period.

Here’s a set of objective criteria you can use to determine whether what you’re doing is truly DevOps. You’re really doing DevOps if…

1. You have frequent, rapid release cycles.
2. Deployment to production is fully automated, and you can automatically roll back.
3. You do continuous integration with automated testing at every check-in.
4. You have far more resources devoted to automated testing than manual testing.
5. Your developers, testers, and operations engineers work together.
6. Your developers, testers, and operations engineers have common, business-oriented goals that cover the whole value chain.
7. You’re paying back your technical debt.
8. Management gives the team the authority to make the changes it needs to make—and the team makes those changes.
9. You have feedback mechanisms in place to ensure continuous assessment—and act on that feedback to ensure continuous improvement.

Step by Step Procedure To Setup Cassandra Cluster On Any Linux Distribution

Step by Step procedure to set-up Cassandra Cluster On Any Linux Distribution

1. What is Cassandra : Cassandra is an open source distributed database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure.

2. Cassandra Cluster: A cluster is arranged as a ring of nodes. Clients send read/write requests to any node in the ring; that node takes on the role of coordinator node, and forwards the request to the node responsible for servicing it. A partitioner decides which nodes store which rows. The recommended partitioners assign rows to nodes based on a hash value of the row key. Nodes are assigned tokens that evenly divide the full range of possible hash values.

3. Prerequisites :
Setup iptables, allow below Cassandra ports.
For Centos:
iptables -A INPUT -p tcp -m tcp –dport 9160 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7000 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7001 -j ACCEPT
service iptables restart

For Ubuntu:
iptables -A INPUT -p tcp -m tcp –dport 9160 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7000 -j ACCEPT
iptables -A INPUT -p tcp -m tcp –dport 7001 -j ACCEPT
sudo sh -c “iptables-save > /etc/iptables.rules”
sudo iptables-apply /etc/iptables.rules
Applying new iptables rules from ‘/etc/iptables.rules’… done.
Can you establish NEW connections to the machine? (y/N) y
… then my job is done. See you next time.

4. Installing Cassandra and Configuring: Login to Remote machine, where you want to install Cassandra.
Below are the steps to install and configure Cassandra Cluster.

Get Cassandra tar-ball
wget “http://downloads.datastax.com/community/dsc-cassandra-2.1.7-bin.tar.gz” -P /usr/share/
Setup Cassandra environment under /usr/share
tar -xvzf /usr/share/dsc-cassandra-2.1.7-bin.tar.gz -C /usr/share/

Delete tar file
rm -rf /usr/share/dsc-cassandra-2.1.7-bin.tar.gz

Cassendra configuration change cluster_name : change clustername with your Cluster
sed -i ‘s/Test Cluster/{{clustername}}/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change listen_address : hostname=serverip
sed -i ‘s/listen_address\:\ localhost/listen_address\:\ {{hostname}}/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change rpc_address
sed -i ‘s/rpc_address\:\ localhost/rpc_address\:\ 0.0.0.0/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change endpoint_snitch
sed -i ‘s/endpoint_snitch\:\ SimpleSnitch/endpoint_snitch\:\ GossipingPropertyFileSnitch/g’ /usr/share/dsc-
cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change broadcast_rpc_address hostname=serverip
sed -i ‘s/^# broadcast_rpc_address\:\ 1.2.3.4/broadcast_rpc_address\:\ {{hostname}}/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change seeds hostname=serverip seeds=seedsip
sed -i ‘s/- seeds\:\ “127.0.0.1”/- seeds\:\ “{{seeds}},{{hostname}}”/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change data_file_directories
sed -i ‘s/# data_file_directories/data_file_directories/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change /var/lib/cassandra/data
sed -i” ‘107 s/^#//’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change commitlog_directory
sed -i ‘s/# commitlog_directory/commitlog_directory/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change saved_caches_directory
sed -i ‘s/# saved_caches_directory/saved_caches_directory/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change thrift_framed_transport_size_in_mb
sed -i ‘s/thrift_framed_transport_size_in_mb\:\ 15/thrift_framed_transport_size_in_mb\:\ 100/g’ /usr/share/dsc-
cassandra-2.1.7/conf/cassandra.yaml

Cassendra configuration change phi_convict_threshold
sed -i ‘s/# phi_convict_threshold\:\ 8/phi_convict_threshold\:\ 10/g’ /usr/share/dsc-cassandra-2.1.7/conf/cassandra.yaml

Start Cassandra service
nohup sh /usr/share/dsc-cassandra-2.1.7/bin/cassandra

5. Command to check Cassandra cluster:
/usr/share/dsc-cassandra-2.1.7/bin/nodetool status
Address Status State Load Owns Token
113427455640312821154458202477256070484
10.1.1.7 Up Normal 459.27 MB 33.33% 0
10.1.1.8 Up Normal 382.53 MB 33.33% 56713727820156410577229101238628035242
10.1.1.6 Up Normal 511.34 MB 33.33% 113427455640312821154458202477256070484

Shell Script : Auto Enable/Disable Specific Cronjob

Shell Script : Auto Enable/Disable Specific Cronjob

What is Cronjob ?
A cron job is a scheduled task that is executed by the system at a specified time/date.

Sometimes we want to perform some task during specific time(Ex: During production Hours), and once the production time completes, we will be disabling cronjob. this requires manual work.The above manual work, can be automated with this script.

Create two scripts,
1.Auto_enable_cron.sh
2. Auto_disable_cron.sh

Below are both the scripts with cronjob details,

COMMANDS used:

crontab -l | sed ‘/^#.*GIVE-SCRIPT-NAME-HERE.sh/s/^#//’ | crontab –
crontab -l | sed ‘/^[^#].*GIVE-SCRIPT-NAME-HERE.sh/s/^/#/’ | crontab –

cronandscript

cronoutput

In the above image we can see that after running Auto_enable_cron.sh
The script in the cron GIVE-SCRIPT-NAME-HERE.sh is uncommented and after running
Auto-disable_cron.sh the script GIVE-SCRIPT-NAME-HERE.sh is commented again.

Now add scripts to desired time in cronjob.
9 * * * * /home/cronjob/Auto_enable_cron.sh (This will enable GIVE-SCRIPT-NAME-HERE.sh @9)
23 * * * * /home/cronjob/Auto_disable_cron.sh (This will disable GIVE-SCRIPT-NAME-HERE.sh @23)

Enjoy, just now you have automated your task!!!!.

Perform SCP Without Entering Password

In this article, I’ll explain how to perform scp without entering the password using the SSH Public Key authentication. There will be requirements like you would like to transfer files from one server to another with scripts and run it as cronjob. There are two levels of security in the SSH key based authentication. In order for you to login, you need both the private key and the passphrase. Even if one of them is compromised, attacker still cannot login to your account, as both of them are needed to login. This is far better than typical password based authentication, where if the password is compromised, attacker can gain access to the system.

There are two ways to perform ssh and scp without entering the password:

  1. No passphrase. While creating key pair, leave the passphrase empty. Use this option for the automated batch processing. for e.g. if you are running a cron job to copy files between machines this is suitable option.
  2. Use passphrase and SSH Agent. If you are using ssh and scp interactively from the command-line and you don’t want to use the password everytime you perform ssh or scp, I don’t recommend the previous option (no passphrase), as you’ve eliminated one level of security in the ssh key based authentication. Instead, use the passphrase while creating the key pair and use SSH Agent to perform ssh and scp without having to enter the password everytime as explained in the steps below.

Step:1. Verify that local-host and remote-host is running openSSH

[local-host]$ ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8b 04 May 2006

[remote-host]$ ssh -V
OpenSSH_4.3p2, OpenSSL 0.9.8b 04 May 2006

Step:2. Generate key-pair on the local-host using ssh-keygen

[local-host]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/waseem/.ssh/id_rsa):<Hit enter>
Enter passphrase (empty for no passphrase): <Enter your passphrase here>
Enter same passphrase again:<Enter your passphrase again>
Your identification has been saved in /home/waseem/.ssh/id_rsa.
Your public key has been saved in /home/waseem/.ssh/id_rsa.pub.
The key fingerprint is:

The public key and private key are typically stored in .ssh folder under your home directory. In this example, it is under /home/waseem/.sshd. You should not share the private key with anybody.

Step:3. Install public key on the remote-host.

Copy the content of the public key from the local-host and paste it to the /home/waseem/.ssh/authorized_keys on the remote-host. If the /home/waseem/.ssh/authorized_keys already has some other public key, you can append this to the end of it. If the .ssh directory under your home directory on remote-host doesn’t exist, please create it.
[remote-host]$ vi ~/.ssh/authorized_keys 

In simple words, copy the local-host:/home/waseem/.ssh/id_rsa.pub to remote-host:/home/jsmith/.ssh/authorized_keys

Step:4. Give appropriate permission to the .ssh directory on the remote-host.

[remote-host]$ chmod 755 ~/.ssh
[remote-host]$ chmod 644 ~/.ssh/authorized_keys

Step:5. Login from the local-host to remote-host using the SSH key

[local-host]$ <You are on local-host here>

[local-host]$ ssh -l waseem remote-host
Last login: Sat Jun 07 2008 23:03:04 -0700 [remote-host]$ <You are on remote-host here>

Step:6 Verify SCP working without password from local server

scp testscpwithoutpassword.csv  waseem@192.168.1.1:/home/waseem

Installing Apache and PHP from source

I’ve just been through the process of installing the latest versions (as of this writing) of Apache 2.4.2 and PHP 5.4.5 from source on a new Debian 6 virtual machine for the first time.

I had some small issues along the way and thought I’d do a short post on the process I went through to get it working and resolve various errors which may help someone out. First we will install Apache, followed by PHP.

Prerequisites

Before starting you will need make, gcc and g++ installed.

apt-get install make gcc g++
I also had the latest version of MySQL installed (5.5.25a), as I wanted to use this later on. The latest version was downloaded from the Dotdeb repository

Installing Apache 2.4.2 from source

Apache was then downloaded from http://httpd.apache.org/download using wget on one of the mirrors in my country and then extracted.

wget http://apache.mirror.uber.com.au//httpd/httpd-2.4.2.tar.gz
tar xf httpd-2.4.2.tar.gz
cd httpd-2.4.2/
./configure –enable-so

Configuring Apache Portable Runtime library …
checking for APR… no
configure: error: APR not found. Please read the documentation.
–enable-so is needed later on for PHP.

At this point I downloaded APR 1.4.6 from apr.apache.org/download using wget and then extracted it.

cd ..
wget http://apache.mirror.uber.com.au//apr/apr-1.4.6.tar.gz
tar xf apr-1.4.6.tar.gz
cd apr-1.4.6/
./configure
make
make install
Apache will also need APR-util installed, which can be downloaded from the apr.apache.org/download page too. When you run configure you will have to specify where the apr config is, you should see this location in the output from make install after finishing apr above.

cd ..
wget http://apache.mirror.uber.com.au//apr/apr-util-1.4.1.tar.gz
tar xf apr-util-1.4.1.tar.gz
cd apr-util-1.4.1/
./configure –with-apr=/usr/local/apr/bin/apr-1-config
make
make install
Now that’s complete we can try Apache again.

cd ../httpd-2.4.2/
./configure –enable-so

checking for pcre-config… false
configure: error: pcre-config for libpcre not found. PCRE is required and available from http://pcre.org/
pcre-8.30.tar.gz was then downloaded from http://pcre.org and for something different, I uploaded it to the server with SFTP to the root directory.

cd ..
tar xf pcre-8.30.tar.gz
cd pcre-8.30/
./configure
make
make install
Now to try Apache again…

cd ../httpd-2.4.2/
./configure –enable-so
make
make install
That’s it, Apache should now be ready to go, let’s move into the default installation and start it up.

cd /usr/local/apache2/bin/
./apachectl start
/usr/local/apache2/bin/httpd: error while loading shared libraries: libpcre.so.1: cannot open shared object file: No such file or directory
ldconfig
./apachectl start
./apachectl -v
Server version: Apache/2.4.2 (Unix)
Server built: Jul 26 2012 20:51:23
That error relating to the shared libraries was resolved by running ldconfig, see the man page on that for more information.

Browsing to the IP address in my web browser now brings up the Apache default page. As ServerSignature is enabled by default, using Firebug for FireFox I can see “Apache/2.4.2 (Unix)” as the Server header in the response headers, confirming the new install of Apache has served my page.

Installing PHP 5.4.5 from source

PHP was downloaded from http://au.php.net/get/php-5.4.5.tar.gz/from/a/mirror and then uploaded to the root directory on the server using SFTP.

I’m configuring PHP with apxs, this wasn’t working for me unless Apache was configured with –enable-so. I was after MySQL support, feel free to leave –with-mysql off.

cd /root
tar xf php-5.4.5.tar.gz
cd php-5.4.5/
./configure –with-apxs2=/usr/local/apache2/bin/apxs –with-mysql=mysqlnd

configure: error: xml2-config not found. Please check your libxml2 installation.
apt-get install libxml2-dev
./configure –with-apxs2=/usr/local/apache2/bin/apxs –with-mysql=mysqlnd
make
make install
php -v
PHP 5.4.5 (cli) (built: Jul 26 2012 21:18:58)
Configuration

Once you have both Apache and PHP installed and running there is a bit of configuration to do so that Apache knows how to use PHP. I created a phpinfo file here /usr/local/apache2/htdocs/phpinfo.php and it will not display in a browser prior to configuration.

First off, let’s copy the library file over to Apache.

cp /root/php-5.4.5/libs/libphp5.so /usr/local/apache2/modules/
Next you will want to edit the Apache configuration file in /usr/local/apache2/conf/httpd.conf and add the following lines into it.

AddType text/html .php
AddHandler php5-script .php
This line should already be in there, referring to the .so file that was previously copied.

LoadModule php5_module modules/libphp5.so
You probably want to edit DirectoryIndex index.html and include index.php after this.

Once complete, restart Apache.

cd /usr/local/apache2/bin/
./apachectl restart