[arch-general] ldconfig -> Aborted.

C Anthony Risinger anthony at xtfx.me
Tue Dec 27 04:35:54 EST 2011


On Thu, Dec 22, 2011 at 4:01 PM, clemens fischer
<ino-news at spotteswoode.dnsalias.org> wrote:
>
>> # ldconfig -v
>> ldconfig: Can't stat /usr/lib64: No such file or directory
>> /usr/lib/libfakeroot:
>>        libfakeroot-0.so -> libfakeroot.so
>> /usr/lib/perl5/core_perl/CORE:
>>        libperl.so -> libperl.so
>> /lib:
>> Aborted
>
> I think there's no harm in "mkdir /usr/lib64".
>
> To me this sounds as if the VM balloons out of memory.  How much RAM is
> allocated to the VM's?

(fair amount of debug output ... summary at end)

yeah i originally tried upping the mem to 1024M+, preventing the
balloon module from loading (since it's an opt-in kernel module), and
not even using mem ballooning -- no changes at all.  the /usr/lib64
stuff isn't a prob, my guess is everyone's machine does that (/lib64
is created by glibc for compat reasons only, not in filesystem
package) ...

... though, after rebuilding glibc with debug syms, i was able to
trace the issue.  `ldconfig` is consistently receiving the correct,
then incorrect(?) inode, twice(!), to an arbitrary library; `ldconfig`
detects this anomaly just before adding the entry to it's aux-cache,
then explicitly calls abort().

while the problem library is `libgcrypt.so.11`, it's not specific to
that lib (if i remove that library, it just fails on a different one)
... possibly a pattern here but not yet sure.

i ran `gdb --args ldconfig -v` (breakpoint, output, backtrace, and
source context provided below):

----------------------------------------------------------------------------
Reading symbols from /sbin/ldconfig...done.
(gdb) break cache.c:620 if (soname!=0x0 && strcmp(soname, "libgcrypt.so.11")==0)
Breakpoint 1 at 0x402e14: file cache.c, line 620.
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
>silent
>printf "\n---- soname: %s\n---- inode:  %i\n---- hash:   %i\n\n", soname, id->ino, hash
>continue
>end
(gdb) run
Starting program: /sbin/ldconfig -v
/sbin/ldconfig: Can't stat /usr/lib64: No such file or directory

---- soname: libgcrypt.so.11
---- inode:  15344348
---- hash:   722

/usr/lib/libfakeroot:
        libfakeroot-0.so -> libfakeroot.so
/usr/lib/perl5/core_perl/CORE:
        libperl.so -> libperl.so
/lib:

---- soname: libgcrypt.so.11
---- inode:  15344350
---- hash:   834


---- soname: libgcrypt.so.11
---- inode:  15344350
---- hash:   834


Program received signal SIGABRT, Aborted.
0x000000000044f4fc in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0  0x000000000044f4fc in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x000000000040c20e in abort () at abort.c:93
#2  0x0000000000402e57 in insert_to_aux_cache (id=0x7fffffffd1b0,
flags=771, osversion=0, soname=0x6db360 "libgcrypt.so.11", used=1) at
cache.c:625
#3  0x0000000000403dea in add_to_aux_cache (stat_buf=<optimized out>,
flags=<optimized out>, osversion=<optimized out>, soname=<optimized
out>) at cache.c:650
#4  0x00000000004023cd in search_dir (entry=0x6d09d0) at ldconfig.c:880
#5  0x0000000000402d09 in search_dirs () at ldconfig.c:1023
#6  main (argc=2, argv=<optimized out>) at ldconfig.c:1372
(gdb) list cache.c:620,625
620       for (entry = aux_hash[hash]; entry; entry = entry->next)
621         if (id->ino == entry->id.ino
622             && id->ctime == entry->id.ctime
623             && id->size == entry->id.size
624             && id->dev == entry->id.dev)
625           abort ();
----------------------------------------------------------------------------

... before adding a new entry to the cache, `ldconfig` loops thru
existing entries and aborts if an *exact* match is found ... and in
this case there appears to somehow be 2 entries to the same library
(with different inodes), the first is bogus (from VM perspective
anyway) and the second is added twice, triggering the abort.

i don't know if v9fs or QEMU is suppose to be changing the inode, but
every file i test is "off by two", example (host/VM, resp):

# stat --format="%i %n" ./lib/libgcrypt.so.11.7.0
15344348 ./lib/libgcrypt.so.11.7.0

# stat --format="%i %n" /lib/libgcrypt.so.11.7.0
15344350 /lib/libgcrypt.so.11.7.0

... `ldconfig` is attempting to add EACH as `libgcrypt.so.11.7.0` (see
gdb output)!  very suspicious.  the first (host?) version is somehow
detected before anything in ld.so.conf.d/* is tried (and gdb confirms
all other libs are found during this period as well) ...

... something is definitely wonky though, because `ldconfig` tries to
add inode `15344348` as `libgcrypt.so.11.7.0`, but that is totally
wrong from guest perspective:

stat --format="%i %n" /lib/l* | grep 15344348
15344348 /lib/libext2fs.so.2.4

... i don't know how the !@#$ it's getting that, but i suspect some
kind of bad interaction between the host/VM page caches, or a bug in
ldconfig, the v9fs kernel module, the "virtfs" server implemented
within QEMU, or possibly something *very* odd about my setup. i'm also
using the "mapped" virtfs option (guest perms/etc are stored in xattrs
on the host) allowing QEMU to run as nobody:kvm instead of root ...
could be part of the problem ... i thought this was the recommended
way, perhaps not.

in conclusion ... the issue is very unlikely to be Arch-specific. i'll
debug a bit more, and take the information to the proper sources, but
i figured i'd do a final update here for closure/interest, but will of
course still gladly accepted any further advice or suggestion.

thanks!

-- 

C Anthony


More information about the arch-general mailing list