xx Dim User’s Manual

Dim User’s Manual

Jean-Paul Chaput

 Jean-Paul.Chaput@lip6.fr

June 2011

Contents

1xIntroduction1.1xGetting Dim2xDim2.1xGoals of Dim2.2xConfiguring

Dim3xyumport4xtl2rpm5xyumsync

1 Introduction

The Dim package provides three tools: Dim, yumport and yumsync. yumsync and Dim are independant but yumport depends on Dim.

Thoses tools are mainly aimed at system administrators with a background in rpm/yum based distributions and rpm building.

Dim and yumport must be run on machine with the same architecture as the distribution. That is 32 bits for i386xand 64 bits for x86_64.

1.1 Getting Dim

Source and binary of dimrpm packages can be downloadeds from here:

11 source  11 dim-1.0.7-1.slsoc6.src.rpm
11 binary  11 dim-1.0.7-1.slsoc6.noarch.rpm

2 Dim

2.1 Goals of Dim

Starting from a medium size computer network, it is often useful to set up a local mirror of the distribution, so not to overload the bandwidth towards the outside of the network and not saturate the public servers. Besides having a complete local copy can come in handy. When setting up the local mirror, the straighforward approach is to do a complete copy. Which means at least two repositories: the base distribution, then the security updates. As the repository holding the security updates contains all the previous outdated version of a package, it can grow big over time. Another aspect is that by keeping the repositories separated you have a "two step" install mode: first step install the base (outdated) distribution, then the updates are applieds. It is now less true because the installer (from SL6) automatically perform the update.

The second mirror approach is to merge the base distribution and it’s updates and keep only the latest version of a package. The result is a distribution behaving like a rollingxone, but only with the official updates.

This is mainly what Dim does: merge repositories and prune outdated packages. In addition, it embbeds createrepo, repoview, and can add gpg signature to packages that are lacking it.

2.2 Configuring Dim

Configuration is stored in /etc/yum/dim.conf. The file is in python format, allowing any legal python construct. The file consist in a series of call to the addRepository()xmethod, which basically tells the repository into which we merge packages and from which repositories.

configuration.addRepository
xxxx(xrepoidxxxxx="sl-local"
xxxx,xoptionHelpx="AppliesxonxSLx6xdistributionxrepository."
xxxx,xcompsxxxxxx="repodata/comps-sl6-%(basearch)s.xml"
xxxx,xrpmSubdirxx="Packages"
xxxx,xupdateRepos=["sl-security"]
xxxx)
xconfiguration.addRepository
xxxx(xrepoidxxxxx="sl-source-local"
xxxx,xoptionHelpx="AppliesxonxSLx6xsourcexdistributionxrepository."
xxxx,xupdateRepos=["sl-source"]
xxxx)
xconfiguration.addRepository
xxxx(xrepoidxxxxx="my-addons"
xxxx,xoptionHelpx="Appliesxonx<MyxAddons>xrepository."
xxxx,xupdateRepos=[]
xxxx)
xconfiguration.addRepository
xxxx(xrepoidxxxxx="my-addons-source"
xxxx,xoptionHelpx="Appliesxonx<MyxAddons>xsourcexrepository."
xxxx,xupdateRepos=[]
xxxx)
xx

11                      addRepository() parameters

11 repoid          11 The target repository id (into which we merge).
11 updateRepos     11 The list of repositories id from which to get the updates.
11 comps           11 A comps file for createrepo and repoview (optional). The %(basearch)sxmay be used to be architecture independant.

11 rpmSubDir       11 By default, Dim copies rpm in the same relative location as in the source repositories. But sometimes we need to insert an extra path, like in the base distribution.

11 optionHelp      11 The help message that will be displayed by Dim when help is requested

11

On the machine hosting the distribution and on which dim is run, there must be two sets of repositories.

The normalxrepositories, used for the host maintenance.

The repositories used by Dim to update the distribution.

Those two different sets must not be mixeds. To achieve the separation, Dim changes the root of the filesystem hierarchy from which yum operates. The Dim repositories are located in

/home/dim/yum.root/etc/yum.repos.d/. For the same reason, to tune yum behavior
when it’s used by Dim, modify /home/dim/etc/yum.conf.

Using Dim

The Dim command line:

dimx[--repoview]x[--addsign]x[--clean-cache]x[--check-update]x\\
xxxxx<repoid1>x[repoid2x...]
x
xx

The default behavior of Dim is to merge the update and call createrepo on the selecteds repositories.

11 Dim parameters

11 repoid               11 Specifies on which repository to run. This is the repoidxgiven in the configuration file.

11 —repoview            11 Run repoview (not run by default).

11 —check-updates       11 Reports for avalaible updates. Do not merge them.
11 —addsign             11 Look for packages missing a signature and try to add one. This option disable the updating.
11 —clean-cache         11 Clear all cached yum data (equivalent to yum clean all).

11

Note 1:xDim will only look for update of package that are already present in the target repository. That is, if completly new packages appears in the source repository, they will not be added. You have to if manually the first time.

Note 2:xDeprecated packages are not erased. They are moved into a backup directory, /home/dim/backup/<repoid>/ so if something goes wrong they can be restored. This directory should be cleaned from time to time.

Other Uses

Another application is to create local repository that merge vendors rpm. You could for instance create a vendorxrepository that merges the contents of Adobexand

VirtualBox.
3 yumport

yumport is an utility to ease the task of porting packages from sibling distributions like Fedora or third party repositories like ATrpms. yumport is aimed at simplicity, it do not uses chrootedxenvironment or sophisticated scheduled rebuilds and report systems.

yumport is able to recursively pull/install dependencies needed to build a given package. yumport is to perform the more tedious task of porting, but il will not correct the code for you or adjust dependency name changes. Nevertheless, many Fedora packages rebuild without problems under SL 6 and it’s for those that yumport has been created.

Using yumport

The yumport command line:

yumportx--package=<package1>x[--package=<package2>x...]
xx

11 yumport parameters

11 —package        11 The name of the package to be rebuild. Can be used multiple times.

11

If everything goes according to plan, the following cycle will be executed: The source file is pulled from the source repository. The sources are unpacked in the rpmbuild directory trees. The dependency are looked upon. Either they are avalaible, and are installed, or they must be compiled from source. In the later case insert them in the head of the rebuild list and continue. All dependencies are installed, the package is built. Source (if needed) and binary packages are moved into the target repository. Dim is called to update source & target repository.

Some important files and directories for yumport:

/etc/yum/yumport.confx: the configuration file.

/home/dim/SRPMS.cachex: the directory holdind the original source files (downloaded from the source repositories). /home/dim/rpmbuildx: the root directory of the rpmbuild process. /home/dim/rpmmacrosx: an on the fly generated file to configure rpmbuild.

/home/dim/yum.root/etc/yum.repos.d/x: the alternate yum directory

where yumport get it’s repositories (shared with Dim).

Configuring yumport

First part: Repository Configuration.

defxguessFedoraDistTagx(xreleasex):
xxxxxmx=xre.match(r".*\.fc(?P<fcversion>\d+)\..*",release)
xxxxxifxm:xreturnx"fc%s"x%x(m.group("fcversion"))
x
xxxxxmx=xre.match(r".*\.(?P<disttag>[^.]+)$",release)x#x$
xxxxxifxm:xreturnxm.group("disttag")
x
xxxxxreturnxrelease
x
xdefxguessPatchedDistTagx(xreleasex):
xxxxxreturnx"slsoc6"
x
xconfiguration.addSourceRepoxxxxxxxx("my-addons-source",guessFedoraDistTag)
xconfiguration.addSourceRepoxxxxxxxx("fedora-14-source",guessFedoraDistTag)
xconfiguration.setTargetRpmsRepoxxxx("my-addons"xxxxxxx,"RPMS")
xconfiguration.setTargetSrpmsRepoxxx("my-addons-source")
xx

11 yumport configuration

11 addSourceRepo        11 Add a source repository (by id). The second parameter must be a fonction indicating how to transform the %dist tag (see below).

11 setTargetRpmsRepo    11 The target binary repository, where the rebuild packages are to be put. The second parameter gives a subdirectory inside the repository.

11 setTargetSrpmRepo    11 The target source repository.

11

In order to keep track of the original source of a package, we customize the %distxtag thanks to the guessFedoraDistTag() function. The convention implemented here is to add up the target distribution tag (slsoc6) with the source one (say: fc14), the result being .slsoc6.fc14. You may implement any convention you like.

Second part: Dependency Resolution.

defxbinaryToSourcex(xbinaryx):
xxxxxdefxrePerlSeparx(xmox):
xxxxxxxxxifxxxmo.group(0)x==x'(':xreturnx"-"
xxxxxxxxxelifxmo.group(0)x==x')':xreturnx""
xxxxxxxxxelifxmo.group(0)x==x"::":xreturnx"-"
xxxxxxxxxreturnx""
x
xxxxxmatchedx=xTrue
xxxxxsourcexx=xbinary
xxxxxifxxxbinary.startswith("libXvMCW"):xxxxxxxxxxxxxsourcex=xbinary
xxxxxelifxbinary.startswith("qt-webkit"):xxxxxxxxxxxxsourcex=x"qtwebkit"
xxxxxelifxbinaryx==x"timidity++-patches":xxxxxxxxxxxxsourcex=x"fluid-soundfont"
xxxxxelifxbinaryx==x"soundfont-utils":xxxxxxxxxxxxxxxsourcex=x"gt"
xxxxxelifxbinaryx==x"gnu.regexp":xxxxxxxxxxxxxxxxxxxxsourcex=x"gnu-regexp"
xxxxxelifxbinaryx==x"msv-msv":xxxxxxxxxxxxxxxxxxxxxxxsourcex=x"msv"
xxxxxelifxbinaryx==x"msv-xsdlib":xxxxxxxxxxxxxxxxxxxxsourcex=x"msv"
xxxxxelifxbinaryx==x"jakarta-commons-configuration":xsourcex=x"apache-commons-configuration"
xxxxxelifxbinaryx==x"jaxp":xxxxxxxxxxxxxxxxxxxxxxxxxxsourcex=x"xml-commons-apis"
xxxxxelifxbinaryx==x"xfce4-doc":xxxxxxxxxxxxxxxxxxxxxsourcex=x"xfce-utils"
xxxxxelifxbinary.startswith("libgoom2"):xxxxxxxxxxxxxsourcex=x"goom2k4"
xxxxxelifxbinary.startswith("bitstream-vera"):xxxxxxxsourcex=x"bitstream-vera-fonts"
xxxxxelifxbinary.startswith("netbeans-platform"):xxxxsourcex=x"netbeans-platform"
xxxxxelifxbinary.startswith("perl("):
xxxxxxxxxprintxsource
xxxxxxxxxsourcex=xre.sub(r"\(|\)|::",rePerlSepar,source)
xxxxxelse:
xxxxxxxxxmatchedx=xFalse
xxxxxreturnx(source,matched)
x
x#xPutxthexlookupxfonctionxinxplace.
xconfiguration.setBinaryToSourceHook(binaryToSource)
xx

While a package is not yet in the yum or rpm database, we cannot guess what it provides. The binaryToSourcexfunction is a work around for this problem. It behave like a lookup table, returning the right source package name for a give dependency. Most of the time, the name of the dependency match the name of the source package, but when it’s not the case, we put an explicit translation. The function shown above has been successfuly tested to recompile mplayerxand vlcxfrom ATrpms.

Third part: tricky packages.

configuration.addExtraInfos("lash"xxx,requires=["libuuid-devel"])
xconfiguration.addExtraInfos("lirc"xxx,buildsxx=[[]x,["--define",x"kmdl_userlandx1"]])
xconfiguration.addExtraInfos("sdcc"xxx,buildsxx=[["--define",x"__stripx/bin/true"]])
xconfiguration.allowUpgrades(["libvpx",x"libvpx-devel"])
xx

11                      addExtraInfos() parameters

11 —requires       11 A supplemental dependency which is missing in the spec file.
11 —builds         11 A list of list. Allows to perform multiple builds with differents macro arguments. For instance lirc will be built twice: the first tile without any arguments (hence the []) and the second time with a –define "kmdl_userlan 1" arguments.

11

By default, yumport never override a package already present in the local repositories, but sometimes we have to allow an upgrade (this should be considered very carefully). allowUpgrages() takes a list of package that are alloweds to upgrade distributions ones.

4 tl2rpm

Automatic portage of ctan packages. This script is mostly a Python rewrite of tl2rpm.c from Jindrich Novy. The main difference from the original program are:

Package-oriented approach. Instead of generating all specs files from a complete

     TEXxLive database (texlive.tlpdb), translate <package>.tlpobj.

Fetch archive files from ctan if needed.

Rebuild the rpm package.

Configuration files of tl2rpm. They resides under /etc/yum/tl2rpm/.

11 File  11 Function
11 changelog             11 The changelog section to be inserted in every rpm (same for all).
11 cls.list              11 When looking inside .tex or .cls file for dependency, we cannot known the kind of required file. This list contains all the dependencies that are .cls, otherwise they are considered to be .sty.
11 pkg-CTAN.list         11 The almost whole list of components in the ctan archive. You may feed it to tl2rpm with the —packagesxoption to rebuild the whole archive (took almost two hours on my computer).
11 pkg-SOURCES.map       11 In some rare case, the name of the rpm package do not match the one of the ctan component. The correspondances are given here (rpm name first, ctan component second).
11 req-blacklist.map     11 Some dependency must not be took into account. They are listed in this file.
11 tl2rpm.conf           11 The main configuration file.

11

The tl2rpm command line:

tlrpmx[--quiet]x[--nocache]x[--all-packages]x\\
xxxxxxx[--max-fails=<number>][--report<report_file]x\\
xxxxxxx[--packages=<package_list>]x[package1xpackage2x...]
xx

11 yumsync parameters

11 —quiet          11 Terse display. One line per package.
11 —no-cache       11 Ignore  already  downloaded  archives  in rpmbuild/SOURCES.
11 —max-fails=<>   11 The number of packages allowed to fail before we stop the run.
11 —packages=<>    11 A file containing a list of packages to rebuild. One package per line, comment start by "#".
11 —all-packages   11 Rebuild the whole ctan archive.
11 —keep-cache     11 Keep previously cached datas. The default is to erase everything from a previous run. This is may not be convenient when updating big set of packages.

11 —report=<>      11 Write a report about what has been done in this file.

11

Note on package names: When giving a package name to tl2rpm, remove the texlive-xprefix. Saves some typing...

5 yumsync

Yumsync allows you to keep a computer stricltyxsynchronized on a predefined set of packages. It is different from the distribution-synchronizationxmode of yum. The yum mode ensure that versions of installed packages are keep in sync with the distribution repository, performing downgrade if necessary. It do not modify the set of installed packages (apart from the obsolete mechanism). The task of yumsync is to keep in sync the set of package itself.

Using yumsync

First, a schematic description as how yumsync work to help you interpret the displayed outpout. To avoid running on stalled cached yum datas, the cache is systematically cleared (equivalent to yumxcleanxall, so every time you run yumsync, you will see the download of the repositories metadatas. Then, yumsync performs a first dependency solve run, using an empty rpm database to compute the exact set of packages to be installeds. And finally, it runs a second dependency solve on the system rpm database to align the installed package set on the requested one.

The yumsync command line:

yumsyncx[--conf=<configuration>]x[--profile=<profileid>]x\\
xxxxxxxxx[--nogpgcheck]x[--sync]
xx

11 yumsync parameters

11 —conf           11 Uses an alternate configuration file
11 —profile        11 Force the profile on which to synchronize. If not specified try to match the host name against the host patterns of each profiles

11 —sync           11 Perform the actual synchronization. By default the script is in dry run mode
11 —nogpgcheck     11 Disable GPG checking
11 —keep-cache     11 Keep previously cached datas. The default is to erase everything from a previous run. This is may not be convenient when updating big set of packages.

11

Before performing the first synchronization you are strongly urged to run in dry mode and thorougly checks what packages are to be removed or installeds.

As the aim of yumsync is to synchronize each computer of a network, the configuration file has to be distributed over the network. To achieve this you may uses —confxto point on a configuration file located on a networked file system or the system configuration could be synchronized with rdist.

Defining Profiles — The Configuration File yumsync.conf

The configuration file is in python format, allowing for any python construct to be used. The configuration file define a list of profiles, each profile supplies a list of hostname patterns, a list of enabled repositories and a list of groups/packages to synchronize with.

configuration.addProfilex(xprofileidx=x"lepka"
xxxxxxxxxxxxxxxxxxxxxxxxxx,xhostsxxxxx=x[x"lepka"
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x".*\.soc.lip6.fr"x]
xxxxxxxxxxxxxxxxxxxxxxxxxx,xrepoidsxxx=x[x"sl",x"epel"x]
xxxxxxxxxxxxxxxxxxxxxxxxxx,xpackagesxx=x[x"@base"
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"@core"
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"zsh"
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
xxxxxxxxxxxxxxxxxxxxxxxxxx)
xx

Any number of profile can be defined. Host pattern may be overlaping, in this case, the first to match is used. A complete example of configuration file is given in subsectionx5.

11                        addProfile() parameters

11 profileid       11 An unique identifier for the profile
11 hosts           11 A python list of regular expressions that will be matched against the host name
11 repoids         11 A python list of repository to be enableds while synchronising
11 packages        11 A python list of groups and/or single packages. Names should be kept from the compsxfile, like in kickstart.

11

A full dependency check is performed on the set of requested groups/packages using the yumdepsolve mechanism. So you only have to put in the list the top levelxgroups or individual package. A good starting point is the %package section of anaconda-ks.cfg file left after any installation.

Example of yumsync.conf

#x-*-xPythonx-*-
x#
x#xThexconfigurationxfilexofxyumsync.
x#xIt'sxaxPythonxfilex(*not*xaxmodule).
x#
x#xPackagesxnames/groupxnamesxasxinxyumxcommandxlinex(pattern)xformat.
x
xsl6Desktopx=x[x"@additional-devel",x"@base",x"@core",x"@debugging",x"@basic-desktop"
xxxxxxxxxxxxxx,x"@desktop-debugging",x"@desktop-platform",xx"@desktop-platform-devel"
xxxxxxxxxxxxxx,x"@development",x"@directory-client",x"@eclipse",xx"@emacs",xx"@fonts"
xxxxxxxxxxxxxx,x"@french-support",xxxx"@general-desktop",xxxx"@graphical-admin-tools"
xxxxxxxxxxxxxx,x"@graphics",xxxxxxxx"@input-methods",xxxxxxxx"@internet-applications"
xxxxxxxxxxxxxx,x"@internet-browser",xxx"@java-platform",xxx"@legacy-x",xxxx"@misc-sl"
xxxxxxxxxxxxxx,x"@network-file-system-client",xxxx"@office-suite",xxxxx"@performance"
xxxxxxxxxxxxxx,x"@perl-runtime",xxxxxx"@print-client",xxxxxx"@remote-desktop-clients"
xxxxxxxxxxxxxx,x"@scalable-file-systems",x"@server-platform",x"@server-platform-devel"
xxxxxxxxxxxxxx,x"@tex",xxxxxxxxxxxx"@technical-writing",xxxxxxxxxxxx"@virtualization"
xxxxxxxxxxxxxx,x"@virtualization-client",xx"@virtualization-platform",xx"@web-server"
xxxxxxxxxxxxxx,x"@console-internet",x"@x11"
xxxxxxxxxxxxx#xEndxofxgroups.
xxxxxxxxxxxxxx,x"elrepo-release",x"epel-release",x"adobe-release-i386",x"atrpms-repo"
xxxxxxxxxxxxxx,x"sl-release-notes",xxx"zsh",xxx"xfig",xxx"graphviz",xxxx"ImageMagick"
xxxxxxxxxxxxxx,x"inkscape",xx"latex2html",xx"thunderbird",xx"createrepo",xxx"urlview"
xxxxxxxxxxxxxx,x"rsh",xx"procmail",xx"fetchmail",xx"mutt",xxx"rdesktop",xxx"tigervnc"
xxxxxxxxxxxxxx,x"tigervnc-server",x"gconf-editor",x"vim-X11",x"python-docs",x"qt-doc"
xxxxxxxxxxxxxx,x"libXpm-devel",xx"libXmu-devel",xxx"libXp-devel",xxx"openmotif-devel"
xxxxxxxxxxxxxx,x"rdist",x"screen",x"lm_sensors",x"yum-plugin-versionlock"
xxxxxxxxxxxxx#xEndxofxindividualxpackages.
xxxxxxxxxxxxxx]
x
xconfiguration.addProfilex(xprofileidx=x"lepka"
xxxxxxxxxxxxxxxxxxxxxxxxxx,xhostsxxxxx=x[x"lepka",x"shadock"x]
xxxxxxxxxxxxxxxxxxxxxxxxxx,xrepoidsxxx=x[x"sl",x"elrepo",x"epel"x]
xxxxxxxxxxxxxxxxxxxxxxxxxx,xpackagesxx=xsl6Desktopx+x\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx[x"a2ps"xxxxxxxxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"repoview"xxxxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"fuse-sshfs"xxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"mod_python"xxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"trac"xxxxxxxxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
xxxxxxxxxxxxxxxxxxxxxxxxxx)
x
xconfiguration.addProfilex(xprofileidx=x"sl6-64"
xxxxxxxxxxxxxxxxxxxxxxxxxx,xhostsxxxxx=x[x"sl6-64"x]
xxxxxxxxxxxxxxxxxxxxxxxxxx,xrepoidsxxx=x[x"sl",x"elrepo",x"epel"x]
xxxxxxxxxxxxxxxxxxxxxxxxxx,xpackagesxx=xsl6Desktopx+x\
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx[x"a2ps"xxxxxxxxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"repoview"xxxxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"fuse-sshfs"xxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx,x"mod_python"xxxxxxxxx#xFromx<r:epel>
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
xxxxxxxxxxxxxxxxxxxxxxxxxx)