[arch-releng] [PATCH 1/1] allow to grow devices

[Gerardo Exequiel Pozzi]

On 08/30/2013 06:36 PM, Christian Hesse wrote:
> Gerardo Exequiel Pozzi <vmlinuz386 at yahoo.com.ar> on Fri, 2013/08/30 13:01:
>> On 08/30/2013 06:12 AM, Christian Hesse wrote:
>>> Gerardo Exequiel Pozzi <vmlinuz386 at yahoo.com.ar> on Thu, 2013/08/29 20:01:
>>>> Ignore me, was a bad smart answer, since you want to increse the size of
>>>> the device!
>>>>
>>>> In any case, the semantics of cowfile_size should not be changed. You
>>>> can add another parameter like grow_ro_dev_size=,
>>>
>>> I add a use case for what was completely useless before. (In fact the cow
>>> files got bigger, but the resulting device size stayed the same.) Do you
>>> really want an new parameter for that?
>>> I do not mind, I can change that if desired.
>>
>> Yes using a new parameter should be more clear.
> 
> Done.
> This parameter only takes affect with values greater than 100.
> 

For my first impression (of what name says), not much clear. What is
grow_ro_dev_size=? a percent? In that case a default 100% says: grow
ro_dev in 100% -> double ro_dev_size and 0% -> no changes.


>>>> also should be ensure
>>>> the proper device shutdown and persistence.
>>>
>>> systemd should disassemble the devices, no? What is needed more?
>>
>> No, systemd can not stop this complex setup, indeed we can not use the
>> provided shutdown script by mkinitcpio, we are using a own shutdown script.
> 
> Ah, looks like I missed to add a file to used_block_devices. Is it ok now?
> Hope so... Otherwise I have to take a deeper look at the shutdown script.
> 

Maybe, you need to check if all things stopped properly when switch back
to initramfs on shutdown.


>>> And what do you refer with persistance? Everything should work just fine
>>> given a persistant system is booted with the same parameters.
>>
>> cow_persistent=P | cow_persistent=N
>>
>> For this extension, should not use the same directory.
>>
>> Of course, this "extension" file act as ro-device, will never be
>> modified during runtime.
> 
> I create a directory /run/archiso/growspace. cowspace is not touched by my
> code. Is that ok?

Looking at the code, growspace files is created on rootfs then deleted
on switch-root. Should be on another tmpfs.

But doing in a different way, you can use a dm-zero target instead.
Since you never never write to this block device, is just to say "i have
a much bigger read-only device".

> 
>>>> We are reaching 3.11 and still no "overlayfs" :( (was promised for 3.10,
>>>> 3.2, ...)
>>>
>>> Well, yes...
>>> On the other hand I really like the idea to solve this with device
>>> mapper. ;)
>>
>> If at some point linux implements an overlayfs, and I still alive ;), I
>> want to move back to filesystem solution instead of block-based like now.
> 
> I am fine with that as long as it is upstream linux. ;)

Sure :)

> 
>>> Just thinking about how to integrate the userspace tools and handling for
>>> this. I would like to add a new hook that adds the binary files
>>> (resize2fs, xfs_growfs, btrfs), so anybody can decide whether or not to
>>> include. But where do we call this?
>>
>> I did not see any real usage of this "extension" inside official image.
>> If you are building a custom image build, why not just create a big
>> filesystem from the start?
> 
> Because bigger filesystem has more metadata. Image size would increase for a
> use case I need only very seldom.
> I perfer to add some lines of code, having the possibility to grow device and
> filesystem when needed.
> 

True, but this can be done from another perspective, just create a big
file for filesystem, but use few blocks, since squashfs supports sparse
files.

To do this in the most trivial way is just adding one line to mkarchiso.
Let says: truncate -s 16G ${_fs_img} after "mkfs". (or even better!
truncate -s 2T ${_fs_img} :D)

There is only one downside if cowspace should be stored on filesystems
that does not support sparse files like FAT cowspace_size= is much more
important to be specified now.

This avoid at all this complex logic, what do you think?


> Here is my new patch, this also has some code to actually grow the filesystem.
> 
> From 7609b2ba1b3b876efc4199d9653b389303cd980a Mon Sep 17 00:00:00 2001
> From: Christian Hesse <mail at eworm.de>
> Date: Fri, 30 Aug 2013 21:10:44 +0200
> Subject: [PATCH 1/1] support growing devices and filesystems
> 
> ---
>  archiso/initcpio/hooks/archiso          | 59
> ++++++++++++++++++++++++++++++++- archiso/initcpio/install/archiso_growfs |
> 9 +++++ 2 files changed, 67 insertions(+), 1 deletion(-)
>  create mode 100644 archiso/initcpio/install/archiso_growfs
> 
> diff --git a/archiso/initcpio/hooks/archiso b/archiso/initcpio/hooks/archiso
> index 66a6273..76dcb6e 100644
> --- a/archiso/initcpio/hooks/archiso
> +++ b/archiso/initcpio/hooks/archiso
> @@ -6,11 +6,27 @@ _mnt_fs() {
>      local img_fullname="${img##*/}";
>      local img_name="${img_fullname%%.*}"
>      local dm_snap_name="${dm_snap_prefix}_${img_name}"
> -    local ro_dev ro_dev_size rw_dev
> +    local ro_dev ro_dev_size rw_dev rw_dev_size pad_dev pad_dev_size
>  
>      ro_dev=$(losetup --find --show --read-only "${img}")
>      echo ${ro_dev} >> /run/archiso/used_block_devices
>      ro_dev_size=$(blockdev --getsz ${ro_dev})
> +
> +    if [[ "${grow_ro_dev_size}" -gt 100 ]]; then
> +        pad_dev_size=$((ro_dev_size/100*grow_ro_dev_size - ro_dev_size))
> +        dd of="/run/archiso/growspace/${img_name}.pad" count=0
> seek=${pad_dev_size} &> /dev/null
> +        pad_dev=$(losetup --find --show --read-only
> "/run/archiso/growspace/${img_name}.pad")
> +        echo ${pad_dev} >> /run/archiso/used_block_devices
> +        echo -e "0 ${ro_dev_size} linear ${ro_dev} 0\n${ro_dev_size}
> ${pad_dev_size} linear ${pad_dev} 0" | dmsetup create "ro_${dm_snap_name}"
> +        echo $(readlink -f /dev/mapper/${dm_snap_name})
>>> /run/archiso/used_block_devices +
> +        ro_dev="/dev/mapper/ro_${dm_snap_name}"
> +        ro_dev_size="$((ro_dev_size + pad_dev_size))"
> +        rw_dev_size="${ro_dev_size}"
> +
> +        msg ":: Device ${dm_snap_name} has been grown, remember to grow your
> filesystem!"
> +    fi
> +
>      if [[ "${cowfile_size}" == "100" ]]; then
>          rw_dev_size=${ro_dev_size}
>      else
> @@ -43,6 +59,40 @@ _mnt_fs() {
>      echo $(readlink -f /dev/mapper/${dm_snap_name})
>>> /run/archiso/used_block_devices }
>  
> +# args: name, fs_type
> +_grow_fs() {
> +    local name="${1}"
> +    local fs_type="${2}"
> +    local device="/dev/mapper/${dm_snap_prefix}_${name}"
> +
> +    case ${aitab_fs_type} in
> +        ext*)
> +            if [[ -x /usr/bin/resize2fs ]]; then
> +                msg ":: Growing ${fs_type} filesystem for ${name}"
> +                resize2fs "${device}" >/dev/null
> +            else
> +                msg ":: Not growing ${name}, userspace tools not available."
> +            fi
> +            ;;
> +        xfs)
> +            if [[ -x /usr/bin/xfs_growfs ]]; then
> +                msg ":: Growing ${fs_type} filesystem for ${name}"
> +                xfs_growfs "${device}" >/dev/null
> +            else
> +                msg ":: Not growing ${name}, userspace tools not available."
> +            fi
> +            ;;
> +        btrfs)
> +            if [[ -x /usr/bin/btrfs ]]; then
> +                msg ":: Growing ${fs_type} filesystem for ${name}"
> +                btrfs resize max "${device}" >/dev/null
> +            else
> +                msg ":: Not growing ${name}, userspace tools not available."
> +            fi
> +            ;;
> +    esac
> +}
> +
>  # args: /path/to/image_file, mountpoint
>  _mnt_sfs() {
>      local img="${1}"
> @@ -112,6 +163,7 @@ run_hook() {
>      else
>          cowfile_size=${cowfile_size/%}
>      fi
> +    [[ -z "${grow_ro_dev_size}" ]] && grow_ro_dev_size="100"
>  
>      if [[ -z "${aitab}" ]]; then
>          aitab="/run/archiso/bootmnt/${archisobasedir}/aitab"
> @@ -188,6 +240,10 @@ archiso_mount_handler() {
>      fi
>      mkdir -p "/run/archiso/cowspace/${cow_directory}"
>  
> +    if [[ "${grow_ro_dev_size}" -gt 100 ]]; then
> +        mkdir -p /run/archiso/growspace
> +    fi
> +
>      local aitab_img aitab_mnt aitab_arch aitab_sfs_comp aitab_fs_type
> aitab_fs_size while read aitab_img aitab_mnt aitab_arch aitab_sfs_comp
> aitab_fs_type aitab_fs_size; do [[ "${aitab_img#\#}" != "${aitab_img}" ]] &&
> continue @@ -195,6 +251,7 @@ archiso_mount_handler() {
>          if [[ "${aitab_fs_type}" != "none" ]]; then
>              _mnt_sfs
> "/run/archiso/bootmnt/${archisobasedir}/${aitab_arch}/${aitab_img}.fs.sfs"
> "/run/archiso/sfs/${aitab_img}" _mnt_fs
> "/run/archiso/sfs/${aitab_img}/${aitab_img}.fs" "${newroot}" "${aitab_mnt}"
> +            [[ "${grow_ro_dev_size}" -gt 100 ]] && _grow_fs "${aitab_img}"
> "${aitab_fs_type}" else
>              _mnt_sfs
> "/run/archiso/bootmnt/${archisobasedir}/${aitab_arch}/${aitab_img}.sfs"
> "${newroot}${aitab_mnt}" fi diff --git
> a/archiso/initcpio/install/archiso_growfs
> b/archiso/initcpio/install/archiso_growfs new file mode 100644 index
> 0000000..6334a3c --- /dev/null
> +++ b/archiso/initcpio/install/archiso_growfs
> @@ -0,0 +1,9 @@
> +#!/bin/bash
> +
> +build() {
> +    add_binary btrfs
> +    add_binary resize2fs
> +    add_binary xfs_growfs
> +}
> +
> +# vim: set ft=sh ts=4 sw=4 et:
> 


-- 
Gerardo Exequiel Pozzi
\cos^2\alpha + \sin^2\alpha = 1

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 555 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.archlinux.org/pipermail/arch-releng/attachments/20130830/27c2a2ea/attachment.asc>