Sample Action Plan for Xen Server Memory Upgrade

INTRODUCTION

A memory upgrade can be one of the least expensive methods of improving computer’s performance. Memory is often the largest limiting factor in application deployment

especially in a Virtual Server environment. Additional RAM will provide provisioning of additional guest domains since CPU is barely utilized.

This document will provide information on the steps on upgrading the memory of server, during and after action plans that needs to undertake.

When upgrading the memory on a Linux server, it is very important to understand the existing memory of the system, which will help plan the activity without opening

the server chassis.

SERVER INVOLVED:

Hostname: SERVER1.freelinuxtutorials.com
IP Address: 10.100.1.3
Type: HP Proliant DL380 G5
Hardware Specification:
Processor:
Memory: DDR2 667MHz PC2-5300 Fully-buffered DIMMs
Max: 65536MB or 64GB
Slots: 8 (4 banks of 2) with a maximum of 8GB per slot
Module Pin Count: 240
Hard disk:
Others:
Software Specification:
OS: Red hat Enterprise Linux 5.4 64-bit
Kernel version: 2.6.18-164.el5xen
Important Service/Apps Running: Xen
Use/Function: Secondary Virtual Domain Host (Dom0)
Virtual Guests: Para-virtualized Guests

I. PRE-INSTALLATION PLANS:

The main key to a successful memory upgrade is to know what kind of RAM your motherboard is currently using.

These are the steps that need to take before doing the actual memory upgrade. As mentioned above, we need to know the existing physical RAM of the server which will

help plan the activity without opening the server chassis.

1. Check the Current Total RAM in the system
We need to know how much memory that the server has and this can be obtained using command “free” or from the /proc/meminfo. We need to remotely access the server via

ssh

a. Ssh the server, and type “free”
[root@SERVER1 ~]# free
total used free shared buffers cached
Mem: 5994496 2042716 3951780 0 260740 893756
-/+ buffers/cache: 888220 5106276
Swap: 2031608 80 2031528

b. Verify in the /proc/meminfo

[root@SERVER1 ~]# grep MemTotal /proc/meminfo
MemTotal: 5994496 kB

2. Check the maximum RAM supported by the system

a. Find out what kind of memory is in the machine by consulting the server manual. Look in the “Technical Specifications” section or Quick Specs. Based on its

official HP server page for HP Proliant DL380 G5 server http://h18000.www1.hp.com/products/quickspecs/12477_na/12477_na.html, it has these following memory

specifications:

• Eight (8) sockets for PC2-5300 Fully Buffered DIMMs (DDR2-667)
• Up to 64 GB PC2-5300 Fully Buffered DIMMs (DDR2-667) with 4:1 and 2:1 interleaving available, online spare and mirrored memory capabilities

b. Use the “dmidecode” command. It will query all memory related information from the DMI table of the system.

[root@SERVER1 ~]# dmidecode -t 16
# dmidecode 2.10
SMBIOS 2.4 present.

Handle 0x1000, DMI type 16, 15 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Single-bit ECC
Maximum Capacity: 64 GB
Error Information Handle: Not Provided
Number Of Devices: 8

c. Check available memory slots for expansion

The system has 8 memory slots and maximum capacity of 64GB. We need to figure out how many slots were used and how many slots are still available. Use the command

below to show this information.

[root@SERVER1 ~]# dmidecode -t 17 |grep Size
Size: 4096 MB
Size: No Module Installed
Size: 4096 MB
Size: No Module Installed
Size: 4096 MB
Size: No Module Installed
Size: 4096 MB
Size: No Module Installed

d. Backup important configuration files and Xen images

II. ACTUAL INSTALLATION of MEMORY

These are the step-by-step procedures in performing the actual memory upgrade for Server

1. Inform management and end-users about the scheduled activity via email
2. Once acknowledged, SSH the server and shutdown

[root@SERVER1 ~]#shutdown –h now

3. Unplug everything from that server and put the machine on a clean work surface. Take it out from the rack and make sure no power going to the server.
(Optional) Touch an unpainted part of the case to dissipate any static electricity
4. Remove the cover.
5. Double check the memory to be inserted if it’s the correct or not. It should be original and compatible“DDR2 PC2-5300” and purchased from HP vendor.
6. Insert the module vertically into the DIMM slot (see Fig.1 )
7. Leave the cover off and hook everything up so we can test it.
8. Power on and closely watch the monitor carefully to see any errors if any. **Check the initial BIOS screen and a counter representing memory test. Take note of

the number, 1MB =1024KB.
9. If everything looks ok, shutdown again and put cover back.
10. Hook everything up again and put it back carefully to the its original location in the rack.
11. Power on the machine and observe again the BIOS screen.

*Memory upgrade problems can be determined initially from number of beeps or incorrect display of installed memory. In case of this, we need to double check the memory

is seated correctly.

III. POST INSTALLATION PLANS

1. Listen to audio beep or Power-on Self Test (POST) errors if any. After the OS boots up, verify if the memory count increased by issuing the same command used

on the pre-installation steps.

[root@SERVER1 ~]#free
[root@SERVER1 ~]#cat /proc/meminfo
[root@SERVER1 ~]# dmidecode -t 17 |grep Size

2. Inspect system logs for any possible errors related to the memory upgrade. Use these commands below to conduct inspections and analysis.

[root@SERVER1 ~]#dmesg
[root@SERVER1 ~]#tail –f /var/log/messages
[root@SERVER1 ~]#tail –f /var/log/xen/xend.log

[root@SERVER1 ~]# xm dmesg
__ __ _____ _ ____ _ __ _ _ _ ____
\ \/ /___ _ __ |___ / / | |___ \ / |/ /_ | || | ___| | ___|
\ // _ \ \047_ \ |_ \ | | __) |__| | \047_ \| || |_ / _ \ |___ \
/ \ __/ | | | ___) || |_ / __/|__| | (_) |__ _| __/ |___) |
/_/\_\___|_| |_| |____(_)_(_)_____| |_|\___/ |_|(_)___|_|____/

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 3.1.2-164.el5 (mockbuild@redhat.com) (gcc version 4.1.2 20080704 (R ed Hat

4.1.2-46)) Tue Aug 18 15:50:30 EDT 2009
Latest ChangeSet: unavailable

(XEN) Command line:
(XEN) Video information:
(XEN) VGA is text mode 80×25, font 8×16
(XEN) VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN) EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN) Found 1 MBR signatures
(XEN) Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 – 000000000009f400 (usable)
(XEN) 000000000009f400 – 00000000000a0000 (reserved)
(XEN) 00000000000f0000 – 0000000000100000 (reserved)
(XEN) 0000000000100000 – 00000000cfe54000 (usable)
(XEN) 00000000cfe54000 – 00000000cfe5c000 (ACPI data)
(XEN) 00000000cfe5c000 – 00000000cfe5d000 (usable)
(XEN) 00000000cfe5d000 – 00000000d0000000 (reserved)
(XEN) 00000000e0000000 – 00000000f0000000 (reserved)
(XEN) 00000000fec00000 – 00000000fed00000 (reserved)
(XEN) 00000000fee00000 – 00000000fee10000 (reserved)
(XEN) 00000000ffc00000 – 0000000100000000 (reserved)
(XEN) 0000000100000000 – 000000042ffff000 (usable)
(XEN) System RAM: 16381MB (16775116kB)
(XEN) Xen heap: 13MB (13468kB)
(XEN) Domain heap initialised: DMA width 32 bits
(XEN) Processor #0 7:7 APIC version 20
(XEN) Processor #4 7:7 APIC version 20
(XEN) Processor #2 7:7 APIC version 20
(XEN) Processor #6 7:7 APIC version 20
(XEN) Processor #1 7:7 APIC version 20
(XEN) Processor #5 7:7 APIC version 20
(XEN) Processor #3 7:7 APIC version 20
(XEN) Processor #7 7:7 APIC version 20
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3000.038 MHz processor.
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU0: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Booting processor 1/4 eip 90000
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU1: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Booting processor 2/2 eip 90000
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU2: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Booting processor 3/6 eip 90000
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU3: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Booting processor 4/1 eip 90000
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU4: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Booting processor 5/5 eip 90000
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU5: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Booting processor 6/3 eip 90000
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU6: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Booting processor 7/7 eip 90000
(XEN) VMX disabled by Feature Control MSR.
(XEN) CPU7: Intel(R) Xeon(R) CPU E5450 @ 3.00GHz stepping 06
(XEN) Total of 8 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) Platform timer overflows in 14998 jiffies.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Brought up 8 CPUs
(XEN) I/O virtualisation disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN) elf_parse_binary: phdr: paddr=0xffffffff80200000 memsz=0x2edd48
(XEN) elf_parse_binary: phdr: paddr=0xffffffff804edd80 memsz=0x14a4f0
(XEN) elf_parse_binary: phdr: paddr=0xffffffff80639000 memsz=0xc08
(XEN) elf_parse_binary: phdr: paddr=0xffffffff8063a000 memsz=0x11b50c
(XEN) elf_parse_binary: memory: 0xffffffff80200000 -> 0xffffffff8075550c
(XEN) elf_xen_parse_note: GUEST_OS = “linux”
(XEN) elf_xen_parse_note: GUEST_VERSION = “2.6”
(XEN) elf_xen_parse_note: XEN_VERSION = “xen-3.0”
(XEN) elf_xen_parse_note: VIRT_BASE = 0xffffffff80000000
(XEN) elf_xen_parse_note: PADDR_OFFSET = 0xffffffff80000000
(XEN) elf_xen_parse_note: ENTRY = 0xffffffff80200000
(XEN) elf_xen_parse_note: HYPERCALL_PAGE = 0xffffffff80206000
(XEN) elf_xen_parse_note: FEATURES = “writable_page_tables|writable_descriptor_t ables|

auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel”
(XEN) elf_xen_parse_note: LOADER = “generic”
(XEN) elf_xen_addr_calc_check: addresses:
(XEN) virt_base = 0xffffffff80000000
(XEN) elf_paddr_offset = 0xffffffff80000000
(XEN) virt_offset = 0x0
(XEN) virt_kstart = 0xffffffff80200000
(XEN) virt_kend = 0xffffffff8075550c
(XEN) virt_entry = 0xffffffff80200000
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 0xffffffff8075550c
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000414000000->0000000418000000 (3992871 pages to be al located)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff80200000->ffffffff8075550c
(XEN) Init. ramdisk: ffffffff80756000->ffffffff81037c00
(XEN) Phys-Mach map: ffffffff81038000->ffffffff82ece938
(XEN) Start info: ffffffff82ecf000->ffffffff82ecf49c
(XEN) Page tables: ffffffff82ed0000->ffffffff82eeb000
(XEN) Boot stack: ffffffff82eeb000->ffffffff82eec000
(XEN) TOTAL: ffffffff80000000->ffffffff83000000
(XEN) ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 8 VCPUs
(XEN) elf_load_binary: phdr 0 at 0xffffffff80200000 -> 0xffffffff804edd48
(XEN) elf_load_binary: phdr 1 at 0xffffffff804edd80 -> 0xffffffff80638270
(XEN) elf_load_binary: phdr 2 at 0xffffffff80639000 -> 0xffffffff80639c08
(XEN) elf_load_binary: phdr 3 at 0xffffffff8063a000 -> 0xffffffff80677028
(XEN) Initrd len 0x8e1c00, start at 0xffffffff80756000
(XEN) Scrubbing Free RAM: …done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type \047CTRL-a\047 three times to switch input to Xen).
(XEN) Freed 112kB init memory.
(XEN) mm.c:625:d4 Non-privileged (4) attempt to map I/O space 000000f0
(XEN) mm.c:625:d4 Non-privileged (4) attempt to map I/O space 000000f0
(XEN) mm.c:625:d4 Non-privileged (4) attempt to map I/O space 000000f0
(XEN) mm.c:625:d4 Non-privileged (4) attempt to map I/O space 000000f0

FALLBACK PROCEDURE:

1. Upon confirming and correct analysis several errors or problems based on the system logs, and negative results of the recent activity such as application

crash, OS kernel panic, system errors and performance degradation, a fallback procedure should be done.
2. Inform related parties and management about the problem and request for emergency maintenance/downtime.
3. Shutdown the server
4. Remove the newly-inserted modules and conduct testing to see if RAM is needed for replacement from the vendor.
5. Power on the server and inspect system logs.
6. Continuously monitor the server again to see stability and performance.
7. Schedule for maintenance for another memory upgrade once RAM is confirmed defective and need replacement.