[arch-dev-public] Arch mirror tracking woes

[Dan McGee]

Whoopsies- wrong ML. Please reply here.


On Tue, Jan 20, 2009 at 3:36 PM, Dusty Phillips <buchuki at gmail.com> wrote:
> 2009/1/20 Aaron Griffin <aaronmgriffin at gmail.com>:
>> On Tue, Jan 20, 2009 at 12:04 AM, Geoffroy Carrier
>> <geoffroy.carrier at koon.fr> wrote:
>>> On Tue, Jan 20, 2009 at 05:24, Dan McGee <dpmcgee at gmail.com> wrote:
>>>> When you wake up, feel free to hack at this- it should do something
>>>> similar to what we want.
>>>
>>> http://github.com/gcarrier/automirror/
>>
>> Neat!
>>
>> The only thing I don't like... this is yet-another-mirror-list. We
>> currently have:
>> * pacman-mirrorlist
>> * The list of IPs for rsyncing mirrors
>> * A django DB of mirrors got the download page
>>
>> Any way you can reuse one of these?
>
> There's also a list of mirrors on the wiki. Its a pain to add a new
> mirror it has to be added in four places.

Let's get this stuff fixed. How does this look for a step in the right
direction?

class Mirror(models.Model):
   id = models.AutoField(primary_key=True)
   domain = models.CharField(max_length=255)
   rsyncip = models.IPAddressField()
   country = models.CharField(max_length=255)
   admin_email = models.EmailField(max_length=255, null=True, blank=True)
   def __unicode__(self):
       return self.domain
   class Meta:
       db_table = 'mirrors'

class MirrorProtocol(models.Model):
   id = models.AutoField(primary_key=True)
   protocol = models.CharField(max_length=10, unique=True)

class MirrorUrl(models.Model):
   id = models.AutoField(primary_key=True)
   url = models.CharField(max_length=255)
   protocol = models.ForeignKey(MirrorProtocol, related_name="urls")
   mirror = models.ForeignKey(Mirror, related_name="urls")

1. rsync IP address will be stored right with the mirror. We can
easily run a DB query and script it to generate our list as needed for
our rsync configuration.
2. Mirror <-> URL is no longer a one-to-one mapping. A mirror could
have 1-4 URLs for any of the protocols it supports (FTP, HTTP, RSYNC,
and maybe one day HTTPS).
3. More detailed storage could turn into a Django view that says "give
me a HTTP and FTP mirrorlist for Canada" and boom! mirrorlist is made.
4. pacman-mirrorlist would be an automatically generated package if at
all possible. The caveat would be the current continent/country-based
sorting- any idea if we need to do something with the freeform Country
field to make this easier?
5. (CCed on this email) Gerhard, does this model look flexible enough
to perhaps incorporate mirror check status information in the future?
I would not mind seeing that feature on the main Arch site.
6. It might also be useful to add a "has_isos" boolean field to those
MirrorURLs (or Mirrors? not sure) that have ISOs mirrored. This would
ensure the download page makes sense.

Diff is below from the current model, although this may be quite a bit
less helpful than the above model.

diff --git a/main/models.py b/main/models.py
index b5a2a42..0fe17d5 100644
--- a/main/models.py
+++ b/main/models.py
@@ -52,15 +52,24 @@ class PackageManager(models.Manager):
 class Mirror(models.Model):
    id = models.AutoField(primary_key=True)
    domain = models.CharField(max_length=255)
+    rsyncip = models.IPAddressField()
    country = models.CharField(max_length=255)
-    url = models.CharField(max_length=255)
-    protocol_list = models.CharField(max_length=255, null=True, blank=True)
-    admin_email = models.CharField(max_length=255, null=True, blank=True)
+    admin_email = models.EmailField(max_length=255, null=True, blank=True)
    def __unicode__(self):
        return self.domain
    class Meta:
        db_table = 'mirrors'

+class MirrorProtocol(models.Model):
+    id = models.AutoField(primary_key=True)
+    protocol = models.CharField(max_length=10, unique=True)
+
+class MirrorUrl(models.Model):
+    id = models.AutoField(primary_key=True)
+    url = models.CharField(max_length=255)
+    protocol = models.ForeignKey(MirrorProtocol, related_name="urls")
+    mirror = models.ForeignKey(Mirror, related_name="urls")
+
 class Press(models.Model):
    id = models.AutoField(primary_key=True)
    name = models.CharField(max_length=255)