[tools] mrepo ... questions

[Dag Wieers]

On Wed, 11 Jul 2007, Justin Hochstetler wrote:

> After the release of Fedora 7, I've been fooling with mrepo serving as
> a repository for it (f7-i386 for the moment).  I've noticed some
> places where mrepo (latest svn 5990 (according to the mrepo file
> itself)) has troubles with it:  [Note: I'm using a FC3 and a FC5
> *server* for mrepo .]
> 
> Fedora 7 appears to use sqlite-based metadata, which the latest
> createrepo I found / compiled (0.4.9) can create using the
> "--database" option.  However, this option ("createrepo-options") does
> not appear to be settable for anything but the 'main' section [not per
> dist or per-repo].  And, if you use '--force', even the 'main' opts
> are overridden to just be ' --pretty', which prevents the option from
> making it to createrepo.

Yes, the configuration syntax currently is pretty limited. We have only 
one level to specify options (distribution). We cannot modify on a per 
repository basis, and I prefer not to add more options on a per 
distribution basis because it slowly becomes too complex to understand 
what's going on.

However, my intention was (since we have a configuration file per 
distribution now) to allow to override certain things on a per 
distribution basis from within a different section.


> Fedora 7 also uses a new/non-standard comps.xml file name:
> ("comps-f7.xml") , which doesn't appear to be picked up by the
> auto-finder/copier/linker portion of mrepo, since it isn't
> "comps.xml".

The real problem here is that createrepo is being shipped with changes 
that either break with previous behaviour, or add new stuff that makes no 
sense with older distributions.

This is really frustrating. Fedora developers only think about their own 
latest release and do not consider that we, Enterprise users, have to 
support up to EL 2.1...


> So, I've sort of patched mrepo to work around those, by making
> 'createrepo-options' and 'groupfilename' be 'dist'-level attributes,
> but I was kinda treating my patch attempt as a learn-python project,
> so I'm not sure it's quite up to production level just yet.  When
> people send in patches, do you want them as 'diff -Naud' from a recent
> svn repo version?

Yes, that would be fine.


> There were a couple of other things I noticed that seemed weird or
> less than optimal:
> 
> a. mrepo downloads the rpms, sends an email, then processes the
> metadata.  I've seen the metadata process fail sometimes, but since
> the email has already gone out, it looks like everything succeeded,
> until users can't yum update the next morning.

The email is send as part of the update process. I understand you expected 
it to be different, but it is important to understand that the update 
process (-u) is not the same as the metadata process (-g).

You can run each of these at a seperate points in time.

In fact, if there are problems in the update or metadata stage, these are 
NOT reflected in the update mail simply because the update mail is a 
report for what is new and what is deleted. Nothing more, nothing less.

If you want a failure report, cron should send you that. Or you can read 
it from the console. (if you don't add any -v's).


> b. mrepo allows you to specify isos to be used in each dist.  but, if
> those isos aren't there, it doesn't stop, and if the 'os' repo is
> defined it just starts downloading 2.4GB of stuff (or so) per dist.  I
> modified my development version to error out if it couldn't find a
> requested iso, so it doesn't try to download the internet. (I just
> added a check after the findisos where if isos is false/none/whatever,
> but .iso was specified, it dies).

mrepo cannot know if all the ISOs are there. And even if it would know 
that they were not there, it does not know if it is intentional or not.

That mrepo does not stop is intentional. There is no reason why ISOs would 
suddenly disappear once you have put them in place.

I understand from your report that if it cannot find an ISO, it enables 
the os repository.

Well, why do you have an os repository if you intend to use the ISOs ? 
That would be my first question :)


> c. one of my servers was old-school yam before I upgraded it.  It
> still functions the same, but now the url starts with
> "http://server/mrepo" instead of "http://server/yam".  I just added an
> one-line alias to the httpd mrepo configuration file that points both
> /yam and /mrepo at the same place.  It seems to work, and my
> already-configured-client machines are happy again.
> 
> "Alias /yam /var/www/mrepo"

The mrepo.conf httpd entry does this for you. It is part of your sysadmin 
job to look at configuration-file changes and merge whatever is relevant. 
I cannot make such decisions for you.
 

> But, those things aside, mrepo is a pretty cool app, and not all the
> stuff I'm thinking about is bugs/etc.

No, it's just that people have different expectations. And if I can clear 
up expectations or make it more straightforward what mrepo is doing I'm 
all interested to make those changes.

But I need people's input (like yours) and maybe suggestions or 
improvements. What would have helped you determine what mrepo was doing at 
some point ? How can I correct wrong expectations for your use-case ?


> Some of the stuff that I am thinking of writing for mrepo:
> 
> (with interspersed questions about the overall design philosophy in a
> couple cases)
> 
> #1
> 
> The mrepo system uses sections for 'dist's and individual line entries
> for 'repo'/url-of-repo entries.  Some of the things I'm thinking of
> trying to add to mrepo would seem to require that repos be definable
> as sections.  For example:
> 
> I have a 'local' repo (actually, I made it a full dist since it's not
> 'release' dependent in my case) to hold things like nxserver,
> vmware-server, etc, that are mostly available as binary RPMs for a
> collection of distros (fc3..7, for example) in one rpm.  I wanted to
> gpg sign those rpms using a my-lan local gpg-key, (which can be done
> easily, manually), but right now, there's not a way to put something
> like 'rpm-signwith=//gpgkeypath'  (for example, as a concept) on a
> single repo, since the repo isn't a section-object like a dist is.
> 
> GPG-Key wise, i'm thinking this would be a good something to be able
> to assign to a repo, so that mrepo could create a web folder of all
> gpg-keys you'd need to import to use any software in the dist/repos.
> Maybe mrepo could also double-check the signatures for each rpm just
> to be double-safe, since users will trust an mrepo repo like it's the
> real thing.

Not sure if this is part of mrepo's job though. mrepo should not 
sign/resign packages and is not a key-manager. What you could do is add a 
package that holds the key and inserts the key. Much like packages as 
rpmforge-release is doing.

That way you can manage keys by updating that package.


> I'm thinking it might also be possible to re-sign (or add a second gpg
> signature to, rather) other 'official' rpms (like from 'extras' or
> 'updates', except that doing that would screw up the sha1 / md5 /
> checksum of the rpms in question, confusing the rsync/etc processes.
> I was thinking of de-sign-ing and re-sign-ing before/after the rsync
> so that they would stay up to date, and I could set my client systems
> to trust a single gpg-key and not have to worry about yum updates
> failing when they either don't have, or want you to approve of, the
> appropriate gpg key for that individual repo.

Signing packages non-interactively is a bad idea. A signature is a 
trust-relation ship, automatically signing stuff you pull in, does not add 
anything to this trust relationship. In fact, it makes things less clear.

If you create your own packages internally, you should sign them yourself. 
With a passphrase ! But anything you do not control, should not be signed 
by your team.


> #2
> 
> I know mirrorlist support is planned/mentioned/to-do-ed for mrepo.  I
> know yum-on-client repo-rpm-generation is as well.  I found an example
> that looks like it might work well to build repo-client-configuration
> rpms for each repo so that they could be yum installed and would then
> enable all the other repos in the repository, etc.  One thing about
> the URLs that mrepo uses is that they're consistent for mrepo, but
> they really tend to look nothing like the various (and sundry) urls
> that the distros tend to use: (for example)
> ".../freshrpms/pub/freshrpms/pub/fedora/fc$release-$arch...etc",
> 
> But, if mrepo allowed you to mangle your local web directory structure
> to match the repo it was mirroring from, you'd be able to change from
> internet to intranet repo usage just by editing the client yum line
> from "downloads.redhat.com" to "mrepo.local.lan" , leaving the path
> alone.  Further, you could do something like a squid proxy and
> lan-DNS-server name-spoof to make 'downloads.redhat.com' resolve to
> point instead to 'mrepo.local.lan', and if the path was 'as expected
> by a virgin client, the client would use your repo automatically, no
> fuss, no muss. :-) [well, assuming mirrorlist support was added, i
> suppose]

I'm not sure I quite follow you here. If you want to match directory 
structures, I would suggest redirecting from a squid proxy to the endpoint 
directories. Although managing the clients is much prefered over trying to 
add some magic (and complexity) on your proxy.

IT environments are not better off with customizations that put 
information on different parts of your environment.


> Okay, this is bit longer than I set out to write, but I think mrepo is
> great, and I think there's definitely some neat things that can be
> added.

I'm open to discuss anything. I'm very interested to move from the current 
config syntax to something yaml based. That would allow us to do more 
complex things.

--   dag wieers,  dag at wieers.com,  http://dag.wieers.com/   --
[Any errors in spelling, tact or fact are transmission errors]