CentOS-Linux4U: Repository: OS Patching on Centos 7

What is OS Patching?

Operating system (OS) patching is an important part of keeping IT systems and applications in your cloud or on-premise environment safe from malicious users that exploit vulnerabilities.

There are 3 different ways of OS patching on centos 7.

OS patching through iso images.
OS patching through the remote repository.
OS patching through vendor repository (Internet).

Now we are going to learn how to patch the centos server.

Note: This complete activity is divided into 6 parts.

We will patch the servers using the iso image.
We will create a remote repository.
We will patch the server using a remote repository and exclude the Apache package.
We will patch the server using the vendor repository and exclude the kernel package.
We will boot the server with an older kernel.
We will remove the old kernel.

Environment :

Application Name: Remote Repository
Host Name: centos-vm1
IP-Address: 192.168.2.201

Application Name: Web Servers
Host Name: centos-vm2
IP-Address: 192.168.2.202

Application Name: Database Server
Host Name: centos-vm3
IP-Address: 192.168.2.203

Pre-requisites :

Copy /etc/host & pre-post-check script file on all server

# scp /etc/hosts centos-vm1:/etc/

# scp /etc/hosts centos-vm2:/etc/

# scp /etc/hosts centos-vm3:/etc/

# scp pre-post-check.sh centos-vm1:/root/

# scp pre-post-check.sh centos-vm2:/root/

# scp pre-post-check.sh centos-vm3:/root/

Attach iso images on all the servers.

# virsh domblklist centos-vm1

# virsh change-media centos-vm1 hda /kvmstore/iso-home/CentOS-7-x86_64-DVD-1810.iso --insert

# virsh change-media centos-vm2 hda /kvmstore/iso-home/CentOS-7-x86_64-DVD-1810.iso --insert

# virsh change-media centos-vm3 hda /kvmstore/iso-home/CentOS-7-x86_64-DVD-1810.iso --insert

# virsh domblklist centos-vm1

Mount nfs mount point & iso image on all the servers.

# yum install net-tools nfs-utils -y

# showmount -e vvm01

# mkdir /nfs-share

# mount vvm01:/nfs /nfs-share/

# mount /dev/cdrom /mnt/

Create virtual ip-address on Web Server

# ip addr add 192.168.2.100/24 dev eth0 label eth0:0

PART 1: LET'S PATCH THE SERVER USING ISO IMAGE.

Login to centos-vm1 and create local repository.

# cd /etc/yum.repos.d/

Move the Centos repository file to backup folder.

# mkdir backup

# mv CentOS-* backup/

Create local repository

# vi local.repo

[Local]

name=Centos-7

baseurl=file:///mnt

enabled=1

gpgcheck=0

Check the repository status.

# yum clean all

# yum repolist

Take the pre-check of the server.

# sh pre-post-check.sh

Broadcast the message on the servers, if someone is performing any activity, he will contact you.

# wall "We are going to patch the server within 15 min and after patching we are going to reboot the servers if you had any concerns kindly contact Vallabh Darole ."

Take a snapshot of centos-vm1 ( If it is a physical server kindly take REAR backup(You can refer to my REAR backup blog.))

# virsh snapshot-list centos-vm1

# virsh snapshot-create centos-vm1

Log in to the console of centos-vm1 and perform the activity.

Note: It is always advisable to perform OS patching activity from the console

Reason: During or After patching the server if the server is not reachable through the network, you can troubleshoot the network issue from console)

# virsh list

# virsh console 1

Check if the packages are available to patch, if the server is already patched you will not see any new packages.

# yum check-update

Patch the server using yum update.

# yum update

Once the patching is completed, reboot the server and boot with the new kernel.

# reboot

Once the server is UP, log in through ssh to the servers and execute the pre-post-check script to see the difference

# sh pre-post-check.sh

Check the output:

# cat /root/Patch_December-2021//Post//DIFFERENCES.TXT | more

OS patching on centos-vm1 is completed

PART 2: CREATE REMOTE REPOSITORY ON CENTOS-VM1

Reason: If you have multiple Centos 7 servers, it will be very difficult to connect iso image to each and every server, hence we suggest creating a remote-repository server

Login to centos-vm1 & mount the iso image.

# mount /dev/cdrom /mnt/

Install vsftp and createrepo package.

# yum install vsftpd createrepo -y

Enable and start the ftp service.

# systemctl enable vsftpd ; systemctl start vsftpd ; systemctl status vsftpd

Create repo directory on the below location for storing packages.

# mkdir /var/ftp/pub/repo

Copy all the packages from iso image to the repo directory.

# cp -a /mnt/ /var/ftp/pub/repo

Create repository using createrepo command.

# createrepo -v /var/ftp/pub/repo/

Remote Repository creation is completed

Part 3: PATCH THE SERVER USING REMOTE REPOSITORY.

Login to centos-vm2

Move the Centos repository file to backup folder.

# cd /etc/yum.repos.d/

# mkdir backup

# mv CentOS-* backup/

Create the remote repo file.

# vi remote.repo

[Remote-Repo]

name=Centos-7

baseurl=ftp://192.168.2.201/pub/repo/

enabled=1

gpgcheck=0

Check the repository status.

# yum clean all

# yum repolist

Take the pre-check of the server.

# sh /root/pre-post-check.sh

Inform the Application team to stop Web Application and after the application is stopped perform the patching.

Broadcast the message on the servers, if someone is performing any activity, he will contact you.

# wall "We are going to patch the server within 15 min and after patching we are going to reboot the servers if you had any concerns kindly contact Vallabh Darole ."

Take a snapshot of centos-vm1 ( If it is a physical server kindly take REAR backup(You can refer to my REAR backup blog.))

# virsh snapshot-list centos-vm1

# virsh snapshot-create centos-vm1

Log in to the console of centos-vm2 and perform the patching.

Note: It is always advisable to perform OS patching activity from the console

Reason: During or After patching the server if the server is not reachable through the network, you can troubleshoot the network issue from console)

# virsh list

# virsh console 1

Check if the packages are available to patch, if the server is already patched you will not see any new packages.

# yum check-update

Patch the server using the yum update command

The application team has informed us not to update Apache packages, hence we will exclude Apache packages.

# yum update --exclude=httpd*

Once the patching is completed, reboot the server and boot with new kernel.

# reboot

Once the server is UP, log in through ssh to the servers and execute the pre-post-check script to see the difference

# sh pre-post-check.sh

Check the output:

# cat /root/Patch_December-2021//Post//DIFFERENCES.TXT | more

Inform Application team to start Web Application

OS patching on centos-vm2 using a remote repository is completed.

Part 4: PATCH THE SERVER USING VENDOR REPOSITORY (INTERNET).

Note: In this scenario, the servers are directly connected to the Internet. which is not recommended.

If you still want to use vendor repository (since it gets updated regularly). Kindly allow your server to connect to vendor portal ONLY.

You can connect to the centos repository directly, but for redhat you need to connect using subscription manager.

Check the repository status.

# yum clean all

# yum repolist

Take the pre-check of the server.

# sh /root/pre-post-check.sh

Inform database team to stop the database and after the database is stopped start the patching.

Broadcast the message on the servers, if someone is performing any activity, he will contact you.

# wall "We are going to patch the server within 15 min and after patching we are going to reboot the servers if you had any concerns kindly contact Vallabh Darole ."

Take a snapshot of centos-vm3 ( If it is a physical server kindly take REAR backup(You can refer to my REAR backup blog for more details.))

# virsh snapshot-list centos-vm3

# virsh snapshot-create centos-vm3

Log in to the console of centos-vm3 and perform the patching.

Note: It is always advisable to perform OS patching activity from the console

Reason: During or After patching the server if the server is not reachable through the network, you can troubleshoot the network issue from console)

# virsh list

# virsh console 1

Check if the packages are available to patch, if the server is already patched you will not see any new packages.

# yum check-update

Patch the server using the yum update command

The database team has informed us not to update the kernel, hence we will exclude the kernel packages.

# yum update --exclude=kernel

Once the patching is completed, reboot the server and boot with the new kernel.

# reboot

Once the server is UP, log in through ssh to the servers and execute the pre-post-check script to see the difference

# sh pre-post-check.sh

Check the output:

# cat /root/Patch_December-2021//Post//DIFFERENCES.TXT | more

Inform database team to start database

OS patching on centos-vm3 using vendor repository is completed.

PART 5: REMOVE THE OLDEST KERNEL FROM THE SYSTEM.

Note: If there are more than 3 kernels you can remove the oldest kernel.

Login to console of centos-vm1 for removal of the older kernel.

# virsh list

# virsh console 1

Check the current running kernel version

# uname -a

Check kernel packages and REMOVE THE OLDEST VERSION only.

# rpm -qa | grep kernel

# yum remove kernel-3.10.0-327.el7.x86_64

Reboot the server.

# reboot

Once the server is UP, Check the current running kernel version

# uname -a

Check kernel packages.

# rpm -qa | grep kernel

PART 6: BOOT THE SERVER USING OLDER KERNEL.

Note: Sometime after completing OS patching, the application does not start. Hence you can boot the server with an older kernel and check.

If still it doesn’t work, restore the snapshot.

Log in to console of centos-vm2

# virsh list

# virsh console 1

Check the current running kernel version

# uname -a

List the Kernel Entries as per GRUB2 file:

# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

Let change booting sequence to older kernel.

# grub2-set-default 1

Changes to /etc/default/grub require rebuilding the grub.cfg file as follows:

# grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the server.

# reboot

Once the server is UP, Check the current running Kernel Version

# uname -a

pre-post-check script.

===============

[root@centos-vm1 ~]# cat pre-post-check.sh

###############START OF SCRIPT#######################

#!/bin/bash

# Added files "/boot/grub/menu.lst" and /etc/SuSE-release

# CHECK THE PACKAGE INSTALLATED FOR THIS SCRIPT.

if rpm -qa | grep nfs-utils >/dev/null ; then echo '' ; else echo 'kindly install nfs-utils package ' ; exit ; fi

if rpm -qa | grep net-tool >/dev/null ; then echo '' ; else echo 'kindly install net-tool package' ; exit ; fi

echo

echo "1. Pre-checks"

echo "2. Post-checks"

echo "3. Quit"

echo "Please select the option:"

read OPT

ROOT_DIR=/root/Patch_`date +"%B-%Y"`

PRE_DIR=$ROOT_DIR/Pre/

BACKUP=$ROOT_DIR/Backup/

POST_DIR=$ROOT_DIR/Post/

BOND_FILE=/proc/net/bonding/bond0

prechecks()

{

if [ -d "$ROOT_DIR" ]; then

echo "$PRE_DIR already exists!!, Please rename it before executing the script"

exit

mkdir $ROOT_DIR

mkdir $PRE_DIR

mkdir $BACKUP

#Files to be backed up

cp -p /etc/redhat-release $BACKUP

cp -p /etc/fstab $BACKUP

cp -p /etc/grub2.cfg $BACKUP

df -PTh > $BACKUP/df.txt

ifconfig -a > $BACKUP/ifconfig_all.txt

mount > $BACKUP/mount.txt

sysctl -a > $BACKUP/sysctl.txt

#Outputs of Important commands

for i in ` ifconfig -a | grep mtu | awk '{print $1}'`;

ethtool $i > $PRE_DIR/ethtool_$i.txt;

done

df -PTh | egrep -vi "type|devtmpfs|tmpfs" > $PRE_DIR/df.txt

uname -a > $PRE_DIR/uname.txt

hostname -I > $PRE_DIR/ifconfig_all.txt

route -n > $PRE_DIR/routing_table.txt

mount | egrep -i "xfs|nfs4|cifs" > $PRE_DIR/mount.txt

netstat -nuatp | grep LIST | awk -F "/" '{ print $2 }' | sort | uniq > $PRE_DIR/services.txt

rpm -qa | sort > $PRE_DIR/rpm.txt

echo "Required files and command outputs copied to $PRE_DIR.."

}

postchecks()

{

if [ -d "$POST_DIR" ]; then

echo "$POST_DIR already exists!!, Please rename it before executing the script"

exit

if [ ! -d "$PRE_DIR" ]; then

echo "Precheck Directory $PRE_DIR does not exist, please perform Prechecks first"

exit

mkdir $POST_DIR

#Files to be backed up

if [ -f "$BOND_FILE" ]; then

cp /proc/net/bonding/bond0 $POST_DIR/bond0.txt

#Outputs of Important commands

for i in ` ifconfig -a | grep mtu | awk '{print $1}'`;

ethtool $i > $POST_DIR/ethtool_$i.txt;

done

df -PTh | egrep -vi "type|devtmpfs|tmpfs" > $POST_DIR/df.txt

uname -a > $POST_DIR/uname.txt

hostname -I > $POST_DIR/ifconfig_all.txt

route -n > $POST_DIR/routing_table.txt

mount | egrep -i "xfs|nfs4|cifs" > $POST_DIR/mount.txt

netstat -nuatp | grep LIST | awk -F "/" '{ print $2 }' | sort | uniq > $POST_DIR/services.txt

rpm -qa | sort > $POST_DIR/rpm.txt

for i in `ls -l $POST_DIR | grep ^- | awk '{print $9}'`

echo >> $POST_DIR/DIFFERENCES.TXT

echo $i >> $POST_DIR/DIFFERENCES.TXT

echo "--------" >> $POST_DIR/DIFFERENCES.TXT

diff -y --suppress-common-lines $PRE_DIR/$i $POST_DIR/$i >> $POST_DIR/DIFFERENCES.TXT

echo >> $POST_DIR/DIFFERENCES.TXT

done

echo

echo "Post Patching Checks done, please check $POST_DIR/DIFFERENCES.TXT for important differences"

HOSTNAME=`hostname`

# echo "$HOSTNAME" | mailx -s "Post-Checks of $RHOSTNAME" -a "$POST_DIR/DIFFERENCES.TXT"

}

case $OPT in

"1") prechecks

;;

"2") postchecks

;;

"3") exit

;;

*) echo "$OPT is not a valid option"

;;

esac

###############END OF SCRIPT#######################

CentOS-Linux4U

Repository: OS Patching on Centos 7

No comments:

Post a Comment