[arch-dev-public] Xen domU kernel - what to do?

[Dan McGee]

On Fri, Oct 30, 2009 at 7:22 PM, Aaron Griffin <aaronmgriffin at gmail.com> wrote:
> On Fri, Oct 30, 2009 at 6:51 PM, Dan McGee <dpmcgee at gmail.com> wrote:
>> On Thu, Oct 29, 2009 at 7:59 PM, Thomas Bächler <thomas at archlinux.org> wrote:
>>> Okay, it is obvious that we need a kernel which can run on Xen domU for our
>>> servers. It would also be very convenient for those running Arch on hosting
>>> services that have pv-grub available.
>>>
>>> Until now, I compiled a modified version of the kernel26 package with
>>> pv_ops/Xern enabled which - in theory - should also run on a real machine.
>>>
>>> If all the external modules compile properly (which they probably do), we
>>> could add these options to our normal stock kernel. I haven't tried running
>>> it on a real machine though and tpowa said in the past the nvidia module
>>> failed. This has to be tested.
>>>
>>> However, compiling Xen on a 32 bit kernel requires enabling PAE which we
>>> refused to do in the past for performance reasons (no idea if these are
>>> valid concerns) and due to the fact that we want to encourage people to run
>>> 64 bit if they have large amounts of memory.
>>>
>>> Assuming all of untested things above work, what would you prefer:
>>>
>>> 1) Switch to PAE on the 32 bit kernel, ship Xen domU support with kernel26
>>> on both architectures.
>>>
>>> 2) Ship Xen domU support on the 64 bit kernel, and don't on the 32 bit
>>> kernel (my favorite so far).
>>>
>>> 3) Leave kernel26 unchanged and create a separate kernel26-xen package with
>>> domU support.
>>>
>>> The advantage of 3) is that the Xen kernel could be very slim, as almost all
>>> hardware drivers could be disabled. The advantage of 1) and 2) is the lower
>>> maintenance cost.
>>>
>>>
>>> On a different note, I'd like to try to switch over to pv-grub on our
>>> servers. Right now, upgrading the kernel requires copying the kernel to the
>>> dom0 manually, thus dom0 access is required to do a (potentially security
>>> relevant) kernel update. With pv-grub, upgrading the kernel and rebooting
>>> would be handled just like with a normal machine, as the dom0 only loads the
>>> pv-grub image which then runs inside the domU and can load the right kernel
>>> from there.
>>> The pv-grub is only available from Xen 3.3 and later, but we have Xen 3.2 -
>>> I compiled pv-grub from 3.4 though and guess it should work with the older
>>> hypervisor as well, but I haven't tested that yet.
>>>
>>> Anyway, please post your opinion on the issues above.
>>
>> I'm -1 on the PAE, leaving options 2 and 3 as viable. 2 is the easiest
>> for maintenance; 3 sounds like the best solution long term as it would
>> be a very lightweight and well-suited kernel package.
>>
>> If not for the maintenance issues, 3 seems by far the best. Is there
>> anyway we can automate the creation of this package to the point where
>> it doesn't take much extra effort? e.g. try to share resources with
>> the primary kernel package.
>
> Can you illustrate why you're -1 on PAE? I can't find exact info on
> the performance hit caused by enabling that, and with machines having
> more and more RAM, I'm sure we'll eventually need to enable PAE. I
> _know_ we're trying to push people to 64bit, but that shouldn't mean
> we leave the i686 arch in the dust.
>
> I'm just sure we're EVENTUALLY going to need to enable this. Why not now?

Completely disagree, PAE is stupid. If you want to stick with 32 bit
and have more memory available, at least run a 64 bit kernel and stick
with a 32 bit userspace:
http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01128.html
http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/01131.html

And those numbers for performance problems? 12%:
http://lkml.indiana.edu/hypermail/linux/kernel/0906.1/00959.html

Leaving i686 in the dust is hardly the concern here; why add
unnecessary overhead to those machines that sit well under the 4GB
mark that need that extra 12% of performance the most?

-Dan