[arch-dev-public] Problem with web dashboard: massive orphaning of packages

[Dusty Phillips]

Hey guys,

I tracked down the problem and fixed it. The basic problem is that
there are two packages in i686 right now that have their arch set to
x86_64:

texlive-core
texlive-htmlxml

ATTENTION: The dbscripts should be updated to ensure that this doesn't
happen. The only valid arches for any db.tar.gz file are the name of
the arch and 'any'.

When reporead found these it was parsing the i686 packages then the
x86_64 packages it found, wiping out any packages from the previous
run. I am guessing the reverse happened in the x86_64 orphaning; in
this case, it was parsing the few i686 packages, which wiped out
x86_64 packages and then processing x86_64 which readded the missing
packages, orphaning them.

I changed it so that if there are packages with the wrong arch, they
are ignored and a warning is printed. Another option would be to
coerce the wrong arch packages to the correct arch for that db.tar.gz,
but I think the optimal solution is to ensure the dbtools don't allow
adding packages with the wrong arch.

The web interface *should* sync up properly with the next running of reporead.

Here's the fix:
http://projects.archlinux.org/?p=archweb_dev.git;a=commitdiff;h=765c6c0cd08973f6cebb007aeff38e82c84e058d;hp=355ca8a7a471094dd4630f669e303da3ff807d6d

Hopefully it should all work now, I didn't test it all that
thoroughly, but it did update my local db.

Hmm, I just realized there's still a chance that 'any' packages from
i686 could clobbel 'any' packages frim x86_64. I didn't look too too
close, but this may be occuring. Aaron if you're bored today you might
want to look into the possibility of that. If not, say hi to Cynthia.

One other thing, I see that the package info files do have references
to the maintainer's name and e-mail. I don't know if these are all
correct or up to date. If they are, I could probably extract the
maintainer info from that and automatically set the package maintainer
instead of setting them to orphan. The problem with this is that if
you orphan a package in the web interface and forget to drop your name
from the maintainer field in the PKGBUILD, you'll get the package
back. I think its safer to stick with the manual way, anybody else
have thoughts on this?

Dusty

2008/9/13 Dusty Phillips <buchuki at gmail.com>:
> 2008/9/13 Aaron Griffin <aaronmgriffin at gmail.com>:
>> On Fri, Sep 12, 2008 at 7:44 PM, Dusty Phillips <buchuki at gmail.com> wrote:
>>> 2008/9/12 Aaron Griffin <aaronmgriffin at gmail.com>:
>>>> They're stored in /tmp/archweb_update.log and emailed to me once a
>>>> day. This is all done in the cron script located at
>>>> /etc/cron.hourly/update_web_db.sh
>>>
>>> What about debug level messages?
>>
>> I'm fairly certain those *don't* go to the syslog, and they're all
>> output to the same script. Maybe the level can be adjusted (at the top
>> it looks like it sets something to WARNING).
>>
>>>> 2008-09-12 18:02:38 -> INFO: Finished repo parsing
>>>> 2008-09-12 18:02:38 -> INFO: Starting database updates.
>>>> 2008-09-12 18:02:38 -> INFO: Updating Arch: x86_64
>>>> 2008-09-12 18:02:47 -> INFO: Finished updating Arch: x86_64
>>>> 2008-09-12 18:02:47 -> INFO: Updating Arch: i686
>>>> 2008-09-12 18:02:47 -> INFO: Removing package kde-l10n-ca from database
>>>> 2008-09-12 18:02:47 -> INFO: Removing package xalan-java from database
>>>> 2008-09-12 18:02:47 -> INFO: Removing package fcgi from database
>>>> 2008-09-12 18:02:47 -> INFO: Removing package enblend-enfuse from database
>>>> 2008-09-12 18:02:47 -> INFO: Removing package netcdf from database
>>>> 2008-09-12 18:02:47 -> INFO: Removing package mirage from database
>>>> 2008-09-12 18:02:47 -> INFO: Removing package glhack from database
>>>> ..... lots and lots of "Removing package" lines ....
>>>> I wonder a) why those were removed and b) if that is related to the
>>>> x86_64 orphaning
>>>
>>> b) is almost certainly yes. The packages get removed and then
>>> presumably get added again later with orphan status. This must be
>>> thoroughly fucking up the web interface new package notification.
>>>
>>> a) is WTF.  I just checked the current state of the db.tar.gz and they
>>> seem to contain packages that reporead claims were removed. So it
>>> doesn't look like anything is breaking the db.tar.gz. It seems more
>>> like reporead is not reading the whole file. But its still possible
>>> the db.tar.gz has been fixed since the error occurred.
>>>
>>> I have added some logging info to say how many packages are currently
>>> in the web db and how many are in the new sync db. If these are
>>> disparate the problem is in the code that loads the repo.db.tar.gz.
>>> Otherwise its in the code that adds/removes packages.
>>>
>>> I also implemented a check to warn or exception if these numbers are
>>> 75% or 50%, as Paul suggested.
>>>
>>> I don't have time to look for anything else right now, hopefully it
>>> will keep happening so I can track it down.
>>>
>>> Does somebody want to give me a quck rundown or wiki article of how
>>> the database tools move packages from svn to release in
>>> repo.db.tar.gz? I'm thinking if reporead wants to be this anal, maybe
>>> we should add some hooks to whatever script says 'I just released a
>>> package, please update the database' and sync up the web database at
>>> the time things get updated.
>>
>> That's actually what we tried to get away from by doing this. The old
>> DB scripts were so tightly coupled to gerolde, it was near impossible
>> to test them. We actually had binaries that did mysql work. I don't
>> want to go back to that way of doing things. This should all be as
>> decoupled as possible....
>>
>>> Sorry I don't know what's causing this folks. I'm just praying its a
>>> long standing bug and can blame it on cactus instead of having to come
>>> back to y'all and say "well here's the thing, I introduced this really
>>> really stupid bug into reporead.py....." ;-)
>>
>> I plan on looking into this on my saturday sprint too. I can do some
>> testing and maybe some improvements of reporead.py too. Should be
>> straightforward - setup a DB, grab the django code, wget the extra DB
>> file, and bam....
>
> Should work, but you'll also need to import a working extra.db.tar.gz
> so that you have the 'correct' packages already in the db. Not sure if
> you have backups or if you can snag one from a mirror somewhere. I
> have a copy of the current broken i686 extra db if it happens to get
> fixed at some point and you want it.
>
> Currently if you run reporead on this db it raises:
>
>  SomethingFishyException: it looks like the syncdb is twice as big as
> the newpackages. WTF?
>
> The problem is hapening before the db_update in reporead.py is called
> and is thus probably in tarfile. The database appears to contain the
> packages.
>
> I'll be working on this for the next hour or so, then I have to give
> up and try to make some money today. Its shaping up to be that day in
> 2008. You know that day every year where absolutely everything goes
> wrong? Its that day for me. My toast landed butter side down this
> morning... So don't expect anything good from me. Anyway, if I don't
> track it down I'll pas it to you with any findings.
>
> Dusty
>