Marco Pracucci

Growing an AWS EBS RAID 0 array, increasing volumes size

by Marco Pracucci Comments

This week @voxnest we had to add more disk capacity to one of our PostgreSQL clusters. Despite it looks a common situation, we never did it so far, because we initially created our clusters with a large RAID 0 array of EBS Volumes. At least, with what we believed to be large enough.

To increase a RAID array made of EBS Volumes you have two main options:

  1. Add more EBS Volumes to the RAID array
  2. Increase the capacity (size) of existing EBS Volumes

Looking for some documentation, we found a lot of documented procedures about #1 while didn’t find much information about #2.

There are pros and cons on both, but to keep it simple we would prefer to not change the number of volumes building up the array because of the automation we have around our PostgreSQL clusters. Our automation currently assumes that all clusters have a RAID 0 composed by the same number of EBS Volumes: adding 1+ volumes only to a specific cluster would require us to revisit and retest it.

All in all, we found out the procedure to be quite simple and I’m going to share it with you in case you will have the same needs.

Pre-requisites

Procedure

  1. Get information about the current RAID (used to compare later - only “Array Size” should change)
    mdadm --detail /dev/RAID-DEVICE
    
  2. Grow EBS Volumes on AWS
    • Modify each EBS Volume size
    • Wait until EBS Volumes change has completed (including the “optimizing” phase)
  3. Drain the instance
    • If you can, drain all production traffic from the instance
  4. Grow RAID
    • Stop service using the RAID (ie. PostgreSQL)

    • Unmount file system
      umount /MOUNT-PATH
      
    • Stop the RAID
      mdadm --stop /dev/RAID-DEVICE
      
    • Re-assemble RAID updating volumes size
      # The key here is --update=devicesize which re-reads the block devices
      # sizes and update the RAID metadata accordingly. Subsequent reassembles
      # of the RAID do NOT need this option anymore.
      
      mdadm --assemble --update=devicesize /dev/RAID-DEVICE LIST-OF-DEVICES
      
    • Ensure the RAID has the new expected size
      mdadm --detail /dev/RAID-DEVICE
      
    • Mount filesystem
      mount -t TYPE /dev/RAID-DEVICE /MOUNT-PATH
      
    • Grow the filesystem size
      # This procedure depends on the filesystem type you're using. The following
      # example shows how it works on XFS.
      
      xfs_info   /MOUNT-PATH
      xfs_growfs /MOUNT-PATH
      xfs_info   /MOUNT-PATH
      
    • Ensure the filesystem has the new expected size
      df -h
      
    • Start service using the RAID (ie. PostgreSQL)
  5. Put the instance back in rotation (only if drained it at step #3)

You may also be interested in ...


Upcoming conferences

I will join the following conferences. Reach me out on Twitter if you wanna meet:
KubeCon Europe Barcelona 20 May 2019 until 23 May 2019
ServerlessDays Milan 21 June 2019

Comments