Lustre Troubleshooting

Hi,

 Last Week, we faced an issue in which our KVM infrastructure, we are using lustre share for storing the disk images of VMs. We have few critical vms running in one of the nodes and those machines get into paused state showing that disk space is full on nodes

Exact error message is given below

"block I/O error in device 'drive-ide0-0-1': No space left on device (28)"

Upon checking the hard drive disk space inside that vm in single usermode, I can see that it had almost 2 TB of free space.

So I believe it was some issue with disk image. So I ran fsck on the disk image in single user mode. But still it didn't worked. Finally I confirmed that it was some issue with the lustre file system, when one more VM running from lustre getinto paused state with the same error.

Finally I began to trouble shoot for lustre errors. Below are the steps that I have done on debugging the lustre

I checked the lustre logs and found the error as


"(vvp_io.c:1018:vvp_io_commit_write()) Write page 369920 of inode ffff8820120601b8 failed -28".

On lustre documentation the error code -28 is "The file system is out-of-space or out of inodes"(http://wiki.lustre.org/manual/LustreManual20_HTML/LustreTroubleshooting.html)


Also I checked the command to list the disk space of all of the OSTs in lustre

lfs df -h 


UUID                       bytes        Used   Available Use% Mounted on
klust01-MDT0000_UUID        1.2T      933.1M        1.1T   0% /klust01[MDT:0]
klust01-OST0000_UUID        9.1T        8.6T      202.8M 100% /klust01[OST:0]
klust01-OST0001_UUID        9.1T        5.8T        2.8T  68% /klust01[OST:1]
klust01-OST0002_UUID        9.1T        5.5T        3.1T  64% /klust01[OST:2]
klust01-OST0003_UUID        9.1T        5.8T        2.9T  67% /klust01[OST:3]
klust01-OST0004_UUID        9.1T        6.5T        2.2T  75% /klust01[OST:4]
klust01-OST0005_UUID        9.1T        6.6T        2.0T  76% /klust01[OST:5]
klust01-OST0006_UUID        9.1T        7.9T      796.0G  91% /klust01[OST:6]
klust01-OST0007_UUID        9.1T        6.4T        2.2T  74% /klust01[OST:7]
klust01-OST0008_UUID        9.1T        6.8T        1.8T  79% /klust01[OST:8]
klust01-OST0009_UUID        9.1T        6.4T        2.2T  74% /klust01[OST:9]
klust01-OST000a_UUID        9.1T        8.6T       48.2M 100% /klust01[OST:10]
klust01-OST000b_UUID        9.1T        7.9T      733.8G  92% /klust01[OST:11]

filesystem summary:       109.1T       82.8T       20.8T  80% /klust01

In above the OST 0 and OST 10 is filledup to 100%.

On checking I can see that both the VM disks are located in OST10  whose disk is filled up.

Below is the command to check in which OST the image locates

lfs getstripe /var/test.img

lmm_stripe_count:   2
lmm_stripe_size:    1048576
lmm_stripe_offset:  8
    obdidx         objid        objid         group
         8           1815776         0x1bb4e0                 0
        10           1728679         0x1a60a7                 0

 So finally we deleted some unused data from ost 10 and then rebooted the OST main node, which fixed the issue.

Cheers 
Syamkumar M 
 
   

How to make a KVM Virtual machine boot From CDROM

Hi,

If you have a running KVM VM and you need to boot this VM from a Bootable Linux iso image inorder to fix some issues on the KVM machine. Below are the steps to boot the KVM machine from CD ROM.

KVM machine is using the xml files for the configuration. We need to edit the configuration xml file inorder to make any changes in the VM.

1_First Shutdown the VM using the command.

virsh shutdown vmname

2)Open the configuration file of the VM using the command

virsh edit vmname

3)Search for the word cdrom and add the lines as below

<disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/opt/isos/CentOS-6.3-x86_64-bin-DVD1.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' unit='0'/>
    </disk>

Here we specified the iso image as the source file parameter.




4)Now the cdrom has been added on the VM. But now we need to make the cdrom as the first boot device.

For that we need to change the boot order of vm. Below is the lines which specifies the boot loader for the KVM vm


<os>
    <type arch='x86_64' machine='rhel6.2.0'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>





In above I put the boot device as cdrom in the parameter <boot dev='cdrom'/> and put it above the hd.

Now reboot the machine and then the vm will boot from the Linux iso image we added.

Cheers
Syamkumar

"Device eth0 does not seem to be present" After Cloning a Centos machine on Vsphere

Hi,

After cloning or moving a machine in VSphere, On starting the Virtual machine, I got error message as below

"Device eth0 does not seem to be present" Delaying initialization.

On cloning a Virtual machine on Vsphere, we need to edit the MAC adress on the VM settings. This change is recognized by the new VM . But it will list the NIC as eth1 instead of eth0, as eth0 MAC adress(Adress of the VM from which we cloned it) is still in the configuration file of new VM.

The configuration file where the Centos machine stores its NIC information is in

  vi /etc/udev/rules.d/70-persistent-net.rule

This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:29:02:bf:b5", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x15ad:0x07b0 (vmxnet3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:50:56:8b:2f:fd", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"


As you can see tat the new MAC Adress is recognized as the adress for eth1. So eth0 won't start.


so we need to edit that file and delete the first entry of in that file.


Then rename the eth1 to eth0  keeping only the second line.


After that start udev configuration using the command below


start_udev

Now restart network services:

service network restart

 

If there is proper configurations are set in the file /etc/sysconfig/network-scripts/ifcfg-eth0, then the eth0 will start.

Cheers

Syam

 

Error: Package: python-paramiko-1.7.6-1.el5.rf.noarch (rpmforge) Requires: python(abi) = 2.4

Hi,

 I had to install a package on a Centos 6.3 machine and I got python dependancy error as below.

Resolving Dependencies
--> Running transaction check
---> Package libedit.x86_64 0:2.11-4.20080712cvs.1.el6 will be updated
---> Package libedit.x86_64 0:20090923-3.0_1.el5.rf will be an update
---> Package libffi.x86_64 0:3.0.5-3.2.el6 will be updated
---> Package libffi.x86_64 0:3.0.9-1.el5.rf will be an update
---> Package perl-Test-Harness.x86_64 0:3.17-129.el6 will be updated
---> Package perl-Test-Harness.noarch 0:3.22-1.el5.rf will be an update
---> Package perl-Test-Simple.x86_64 0:0.92-129.el6 will be updated
---> Package perl-Test-Simple.noarch 0:0.98-1.el5.rf will be an update
---> Package portreserve.x86_64 0:0.0.4-9.el6 will be updated
---> Package portreserve.x86_64 0:0.0.5-2.el5.rf will be an update
---> Package python-beaker.noarch 0:1.3.1-6.el6 will be updated
---> Package python-beaker.noarch 0:1.5.3-1.el5.rf will be an update
--> Processing Dependency: python(abi) = 2.4 for package: python-beaker-1.5.3-1.el5.rf.noarch
---> Package python-mako.noarch 0:0.3.4-1.el6 will be updated
---> Package python-mako.noarch 0:0.3.5-1.el5.rf will be an update
--> Processing Dependency: python(abi) = 2.4 for package: python-mako-0.3.5-1.el5.rf.noarch
---> Package python-markupsafe.x86_64 0:0.9.2-4.el6 will be updated
---> Package python-markupsafe.x86_64 0:0.11-1.el5.rf will be an update
--> Processing Dependency: python(abi) = 2.4 for package: python-markupsafe-0.11-1.el5.rf.x86_64
---> Package python-paramiko.noarch 0:1.7.5-2.1.el6 will be updated
---> Package python-paramiko.noarch 0:1.7.6-1.el5.rf will be an update
--> Processing Dependency: python(abi) = 2.4 for package: python-paramiko-1.7.6-1.el5.rf.noarch
--> Finished Dependency Resolution
Error: Package: python-mako-0.3.5-1.el5.rf.noarch (rpmforge)
           Requires: python(abi) = 2.4
 Installed: python-2.6.6-36.el6.x86_64 (@anaconda-RedHatEnterpriseLinux-201301301459.x86_64/6.4)
               python(abi) = 2.6
           Available: python-2.6.5-3.el6.i686 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.5-3.el6_0.2.i686 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-20.el6.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-29.el6.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-29.el6_2.2.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-29.el6_3.3.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
Error: Package: python-paramiko-1.7.6-1.el5.rf.noarch (rpmforge)
           Requires: python(abi) = 2.4

 Installed: python-2.6.6-36.el6.x86_64 (@anaconda-RedHatEnterpriseLinux-201301301459.x86_64/6.4)
               python(abi) = 2.6
           Available: python-2.6.5-3.el6.i686 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.5-3.el6_0.2.i686 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-20.el6.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-29.el6.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-29.el6_2.2.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
           Available: python-2.6.6-29.el6_3.3.x86_64 (rhel-6-workstation-rpms)
               python(abi) = 2.6
Error: Package: python-beaker-1.5.3-1.el5.rf.noarch (rpmforge)
           Requires: python(abi)

 Installed: python-2.6.6-36.el6.x86_64 (@anaconda-RedHatEnterpriseLinux-201301301459.x86_64/6.4)
               python(abi) = 2.6
     Available: python-2.6.5-3.el6.i686 (rhel-6-workstation-rpms)

The reason is that the python 2.6 package was installed by default on the Centos .But the new installation needs the python 2.4 . The best option is to install python-devel package for that.

So we can install it by enabling the EPEL repository in Centos. Below are the steps to enable EPEL repository in Centos

Enable EPEL Repository

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

Then do the following steps

yum clean all

yum clean metadata

Then install the required package. It will work

Cheers
Syam 

Reset Qmail Admin Postmaster Password

Hi All,


You need to login to Qmail Admin as postmaster inorder to control the email settings or to create new email accounts for the domain etc. But once you forget the password of postmaster account of the domain, you won't be able to manage the domain through the admin toaster.


Below is the step to reset the Admin password for the Qmail

1)First login to the Qmail server
2)Reset the password using the command as below

"/home/vpopmail/bin/vpasswd postmaster@testdomain.com"

change the password to new one. Now you will be able to login to the postmaster account with the new password.

Regards
Syamkumar

no iKVM64 in java.library.path on SuperMicro IP KVM

Hi All,


I faced an issue on accessing IPMI on  ubuntu 11.10. On launching java .jnlp file I am getting error message as below and got disconnected

"no iKVM64 in java.library.path on SuperMicro IP KVM"

A workaround for this problem is to download the .jnlp file and open it with an editor. Copy the two lines stated below, taken from the <resources> entry, to the corresponding <resources os=”Linux” arch=”amd64″> entry. Then start the file with java webstart.

<property name="jnlp.packEnabled" value="true"/>
<property name="jnlp.versionEnabled" value="true"/>

How to Install Perl modules using Cpan

Hi,

Below is the steps to install perl modules using cpan(Comprehensive Perl Archive Network) in a CentOS machine.

The CPAN's main purpose is to help programmers locate modules and programs not included in the Perl standard distribution

It will act as a command line interpretor through which we can install perl modules.

You can get more details about Cpan in the wiki link http://en.wikipedia.org/wiki/CPAN.

sudo  perl -MCPAN -e shell

Above command will drop you in the CPAN shell.


cpan shell -- CPAN exploration and modules installation (v1.9402)
Enter 'h' for help.

cpan[1]> 

Now you can install a perl module. If you want to install a perl
module named Email::Sender, then you need to issue the below command.

cpan[1]> install Email::Sender

That is install module::name(Email is the module here and Sender is the name)

Once the installation shows as succesfull, then you can test it by listing installed perl modules using the command

# instmodsh

Available commands are:
   l            - List all installed modules
   m <module>   - Select a module
   q            - Quit the program
cmd? l
Installed modules are:
   Authen::SASL
   CPAN::Meta::Requirements
   CPAN::Meta::YAML
   Capture::Tiny
   Class::Load
   Class::Method::Modifiers
   Data::OptList
   Data::Serializer
   Devel::GlobalDestruction
   Devel::StackTrace
   Digest::SHA
   Dist::CheckConflicts
   Email::Abstract
   Email::Address
   Email::Date::Format
   Email::MIME
   Email::MIME::ContentType
   Email::MIME::CreateHTML
   Email::MIME::Encodings
   Email::MessageID
   Email::Sender
   Email::Simple
   ExtUtils::CBuilder
   ExtUtils::Config
   ExtUtils::Helpers
   ExtUtils::InstallPaths
   ExtUtils::MakeMaker
   File::Policy
   File::Slurp
   HTML::TokeParser::Simple
   IPC::Cmd
   Import::Into
   JSON::PP
   List::MoreUtils
   Locale::Maketext::Simple
   Log::Trace
   MIME::Tools
   MIME::Types
   MRO::Compat
   Mail::Sender
   Module::Build
   Module::Build::Tiny
   Module::CoreList
   Module::Implementation
   Module::Load
   Module::Load::Conditional
   Module::Metadata
   Module::Runtime
   Moo
   MooX::Types::MooseLike
   Package::Stash
   Params::Check
   Params::Util
   Parse::CPAN::Meta
   Perl
   Perl::OSType
   Role::Tiny
   Sub::Exporter
   Sub::Exporter::Progressive
   Sub::Install
   Sub::Override
   Test::Requires
   Test::Simple
   Test::Tester
   Throwable
   Try::Tiny
   parent
   strictures
   version

That's all . ..............