New directive for %files section (%exclude)

[Philip Prindeville]

I'm new to the list, so apologies if this has already come up.

I was working recently on a .spec file for the proftpd package for the
1.3.1 release, and ran into the following issue.

There are multiple dynamically loadable plug-in modules that can be
built as part of this software package, but not all of them are useful
for all users, and not everyone wants to pull in a kitchen sink-full of
required packages to support it (such as libcap, libwrap, openssl,
krb5-libs, openldap, etc).

So I was thinking that the base package would include the more useful
modules, or the modules that don't have any external dependencies, and
then the remaining modules that require LDAP, Postgres, SSL, MySQL, etc.
couple be splintered into optional sub-packages.

So far so good, right?

Well, then it starts getting complicated, because I have:

%package
...

%files
...
%{_libexecdir}/mod_*.so
...

%package mysql
...
%files mysql
%{_libexecdir}/mod_sql_mysql.so

etc.

And indeed, predictably, /usr/libexec/mod_sql_mysql.so ends up being
included in both packages.

So I was thinking that having a directive like:

%files
...
%exclude %{_libexecdir}/mod_sql_mysql.so
%{_libexecdir}/mod_*.so

would be handy. That is, first all the patterns would be applied and
the list built, then %exclude directives would be applied and the list
whittled down.

This would also be handy in another instance: Fedora doesn't require
the .a and .la files that libtool seems to want to install
unconditionally (grrrr)... and having a script like:

%install
make install DESTDIR=$RPM_BUILD_ROOT
rm -f $DESTDIR/%{_libexedir}/mod_*.{la,a}
...

seems just plain wrong. So the %exclude directive would exclude them
from the package, but also tell RPM, "Yes, these files will be left
behind as artifacts of the 'make install', and I know that, so don't
crap out telling me that files were left over" (or whatever).

Does this seem reasonable, or am I missing something?

-Philip

[Michael Jennings]

On Thursday, 13 September 2007, at 20:34:14 (-0700),

Post by Philip Prindeville
%files
...
%exclude %{_libexecdir}/mod_sql_mysql.so
%{_libexecdir}/mod_*.so
would be handy. That is, first all the patterns would be applied and the
list built, then %exclude directives would be applied and the list whittled
down.
Does this seem reasonable, or am I missing something?

You mean aside from the fact that it already works? :-)

Michael

[Philip Prindeville]

Post by Michael Jennings
On Thursday, 13 September 2007, at 20:34:14 (-0700),

Post by Philip Prindeville
%files
...
%exclude %{_libexecdir}/mod_sql_mysql.so
%{_libexecdir}/mod_*.so
would be handy. That is, first all the patterns would be applied and the
list built, then %exclude directives would be applied and the list whittled
down.
Does this seem reasonable, or am I missing something?

You mean aside from the fact that it already works? :-)
Michael

How's that?

Is it in rpm 4.4.2.1?

I was going by the official version of Maximum RPM... which might not
have been updated to include it yet.

-Philip

[Philip Prindeville]

Post by Philip Prindeville

Post by Michael Jennings
On Thursday, 13 September 2007, at 20:34:14 (-0700),

Post by Philip Prindeville
%files
...
%exclude %{_libexecdir}/mod_sql_mysql.so
%{_libexecdir}/mod_*.so
would be handy. That is, first all the patterns would be applied
and the list built, then %exclude directives would be applied and
the list whittled down.
Does this seem reasonable, or am I missing something?

You mean aside from the fact that it already works? :-)
Michael

How's that?
Is it in rpm 4.4.2.1?
I was going by the official version of Maximum RPM... which might not
have been updated to include it yet.
-Philip

Actually, the Redhat documentation:

http://docs.fedoraproject.org/drafts/rpm-guide-en/

doesn't make any reference to it either.

What is a good and current tutorial on RPM?

-Philip

[Dmitry S. Makovey]

Post by Philip Prindeville
%package
...
%files
...
%{_libexecdir}/mod_*.so
...
%package mysql
...
%files mysql
%{_libexecdir}/mod_sql_mysql.so
etc.
And indeed, predictably, /usr/libexec/mod_sql_mysql.so ends up being
included in both packages.
%files
...
%exclude %{_libexecdir}/mod_sql_mysql.so
%{_libexecdir}/mod_*.so
would be handy. That is, first all the patterns would be applied and
the list built, then %exclude directives would be applied and the list
whittled down.

for that there is this:

%files -f all_but_mysql

%files mysql
%{_libexecdir}/mod_sql_mysql.so

where all_but_mysql could be built using:

find $RPM_BUILD_ROOT | \
grep -v mod_sql_mysql | \
sed -e 's#'$RPM_BUILD_ROOT'##g' > all_but_mysql

it works without overcomplicating things with "exclude". all heavy-lifting
could be done generating approriate file-lists (even if it's done
automatically).

[Michael Jennings]

On Friday, 14 September 2007, at 10:15:05 (-0600),

Post by Dmitry S. Makovey
%files -f all_but_mysql
%files mysql
%{_libexecdir}/mod_sql_mysql.so

There is also %exclude.

Post by Dmitry S. Makovey
find $RPM_BUILD_ROOT | \
grep -v mod_sql_mysql | \
sed -e 's#'$RPM_BUILD_ROOT'##g' > all_but_mysql
it works without overcomplicating things with "exclude".

How on earth do you imagine all that mess is better than one simple
%exclude directive. The method you suggest is the overcomplication,
not the other way around.

Michael

[Dmitry S. Makovey]

Post by Michael Jennings
On Friday, 14 September 2007, at 10:15:05 (-0600),

Post by Dmitry S. Makovey
%files -f all_but_mysql
%files mysql
%{_libexecdir}/mod_sql_mysql.so

There is also %exclude.

Post by Dmitry S. Makovey
find $RPM_BUILD_ROOT | \
grep -v mod_sql_mysql | \
sed -e 's#'$RPM_BUILD_ROOT'##g' > all_but_mysql
it works without overcomplicating things with "exclude".

How on earth do you imagine all that mess is better than one simple
%exclude directive. The method you suggest is the overcomplication,
not the other way around.

in most of RPMs I build I have pretty complicated rules of file
inclusion/exclusion. esp. (Logical AND/OR/XOR) which is easier done with
find/grep/sed then any combination of excludes. Besides it keeps SPEC file
portable for those old systems that run RPM 4.0 (which I have quite a few).

[Michael Jennings]

On Friday, 14 September 2007, at 10:39:19 (-0600),

Post by Dmitry S. Makovey
in most of RPMs I build I have pretty complicated rules of file
inclusion/exclusion. esp. (Logical AND/OR/XOR) which is easier done with
find/grep/sed then any combination of excludes.

But we were talking about *his* situation, not yours, and your
suggestion is clearly wrong for that case.

Post by Dmitry S. Makovey
Besides it keeps SPEC file portable for those old systems that run
RPM 4.0 (which I have quite a few).

Again, not applicable to his situation.

Michael

[Philip Prindeville]

Post by Michael Jennings
On Friday, 14 September 2007, at 10:39:19 (-0600),

Post by Dmitry S. Makovey
in most of RPMs I build I have pretty complicated rules of file
inclusion/exclusion. esp. (Logical AND/OR/XOR) which is easier done with
find/grep/sed then any combination of excludes.

But we were talking about *his* situation, not yours, and your
suggestion is clearly wrong for that case.

Post by Dmitry S. Makovey
Besides it keeps SPEC file portable for those old systems that run
RPM 4.0 (which I have quite a few).

Again, not applicable to his situation.
Michael

Gentlemen! They were both good suggestions, and I thank you equally.

I decided to go with the %exclude suggestion, just because it's "new and
improved", and to give reassurance to whomever might have added it and
then had second thoughts that it was indeed useful... ;-)

The globbing that %exclude uses is somewhat limited, i.e. it doesn't
allow .{a,la} expansions, so in that case, using find/echo with egrep -v
might have been useful... had things gotten any hairier, I would have
probably used a dynamically generated list. Especially if my build were
generating fat binaries (i.e. both i686 and x86_64) and I needed to
triage them...

-Philip

[Michael Jennings]

On Friday, 14 September 2007, at 09:52:05 (-0700),

The globbing that %exclude uses is somewhat limited, i.e. it doesn't allow
.{a,la} expansions

%exclude %{_libdir}/*.*a

Michael

[Philip Prindeville]

Post by Michael Jennings
On Friday, 14 September 2007, at 09:52:05 (-0700),

The globbing that %exclude uses is somewhat limited, i.e. it doesn't allow
.{a,la} expansions

%exclude %{_libdir}/*.*a
Michael

Anyone want to volunteer and have a look at the .RPM I put together for
ProFTPD? Maybe tell me what I could do to make it less Fedora-centric,
or install/uninstall more robustly? If so, email me out-of-band, and
I'll point you at the repository...

Thanks,

-Philip

[Matthew Miller]

Post by Philip Prindeville
Anyone want to volunteer and have a look at the .RPM I put together for
ProFTPD? Maybe tell me what I could do to make it less Fedora-centric,
or install/uninstall more robustly? If so, email me out-of-band, and
I'll point you at the repository...

What's wrong with the proftpd RPM in Fedora?

[Philip Prindeville]

Post by Matthew Miller

Post by Philip Prindeville
Anyone want to volunteer and have a look at the .RPM I put together for
ProFTPD? Maybe tell me what I could do to make it less Fedora-centric,
or install/uninstall more robustly? If so, email me out-of-band, and
I'll point you at the repository...

What's wrong with the proftpd RPM in Fedora?

Besides being several versions old, it doesn't come with a lot of the
useful plug-ins (modules) compiled.

-Philip

[Matthew Miller]

Post by Philip Prindeville
Besides being several versions old, it doesn't come with a lot of the
useful plug-ins (modules) compiled.

1.3.1rc3 is old?

Anyway, sorry, this is really off-topic.

[Philip Prindeville]

Post by Matthew Miller

Post by Philip Prindeville
Besides being several versions old, it doesn't come with a lot of the
useful plug-ins (modules) compiled.

1.3.1rc3 is old?
Anyway, sorry, this is really off-topic.

Last I looked (2 weeks ago), only 1.3.0 was available.

Another issue (perhaps more on topic) is that the single monolithic
build either doesn't provide useful modules, or it pulls in a lot of
modules (with their requisite dependencies) that ends up loading your
system up with a bunch of kruft that you might not want or need.

A better structuring of the RPM would build optional sub-packages that
contain the optional stuff so that it could be individually loaded as
needed.

That's what I've been trying to do. Plus it would be nice to get a
fairly generic .spec file into the source tree, so that people can
actively build off of trunk...

-Philip

[Rodrigo Barbosa]

Post by Philip Prindeville
%{_libexecdir}/mod_*.so

This is such a bad practice I almost cried from reading this e-mail.

NEVER do this. Ever.

If you reuse your specfile for a new version of the package, you
will completely loose (developer) control of which files are in
your package.

Also name all your files.

- --
Rodrigo Barbosa
"Quid quid Latine dictum sit, altum viditur"
"Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)

[Michael Jennings]

On Tuesday, 18 September 2007, at 12:43:16 (-0300),

Post by Rodrigo Barbosa

Post by Philip Prindeville
%{_libexecdir}/mod_*.so

This is such a bad practice I almost cried from reading this
e-mail.

If that's the "worst" thing in his spec file, he's off to a great
start.

Post by Rodrigo Barbosa
NEVER do this. Ever.

There are numerous good reasons to do something like this. I won't go
into them, however.

Post by Rodrigo Barbosa
If you reuse your specfile for a new version of the package, you
will completely loose (developer) control of which files are in your
package.

An interesting, but false, claim.

If one is such a fascist packager that one insists on explicit naming
of every file to be included in a package, one should probably ask
oneself at what point one's dogma has ceased to be an asset.

The purpose of packaging is to organize and manage, not to dictate.
If you are so concerned with the changes occuring between releases of
an upstream package, perhaps you shouldn't be redistributing that
package.

Michael

[Rodrigo Barbosa]

Post by Michael Jennings
The purpose of packaging is to organize and manage, not to dictate.
If you are so concerned with the changes occuring between releases of
an upstream package, perhaps you shouldn't be redistributing that
package.

I won't get into a spit-fight with you over this, or try to offend you
as you tried (without any success) to offend me with namecalling and
stuff.

Try working a Linux distribution lab, and having to manage 80
different packages, and you will understand my point.

If you are only managing 5 or 6 packages, using glob for files
will be almost never an issue. Or if you are the software developer
and is packaging your own software. However, good habits should be
cultivated.

Unless, of course, you don't care.

- --
Rodrigo Barbosa
"Quid quid Latine dictum sit, altum viditur"
"Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)

[Michael Jennings]

On Wednesday, 19 September 2007, at 11:06:23 (-0300),

Post by Rodrigo Barbosa
I won't get into a spit-fight with you over this, or try to offend
you as you tried (without any success) to offend me with namecalling
and stuff.

I'm afraid you missed the point. I was not attempting to offend. I
was drawing an analogy between fascist dictatorships and dictatorial
packaging "rules."

Post by Rodrigo Barbosa
Try working a Linux distribution lab, and having to manage 80
different packages, and you will understand my point.

Oh, I see. Yes, I'm sure you're right.

LOL,
Michael

[Dmitry S. Makovey]

Post by Rodrigo Barbosa

Post by Philip Prindeville
%{_libexecdir}/mod_*.so

This is such a bad practice I almost cried from reading this e-mail.
NEVER do this. Ever.
If you reuse your specfile for a new version of the package, you
will completely loose (developer) control of which files are in
your package.
Also name all your files.

Out of pure curiosity (and for the sake of flaming at all) could you please
give some usecases when such technique (described by Philip P.) backfires?
Building packages myself I am very interested in a ways to improve packaging.

For example: some packages I built contain enormous number of files (3K+) -
what do you do then? Obviously you can't enumerate all of them and it's hard
to track every single change even between upstream revisions (changed 10
filenames etc.)

If you feel more comfortable replying off-list please do so - either way I'm
interested in your opinion. thanks.

[Rodrigo Barbosa]

Post by Dmitry S. Makovey

Post by Rodrigo Barbosa

Post by Philip Prindeville
%{_libexecdir}/mod_*.so

This is such a bad practice I almost cried from reading this e-mail.
NEVER do this. Ever.
If you reuse your specfile for a new version of the package, you
will completely loose (developer) control of which files are in
your package.
Also name all your files.

Out of pure curiosity (and for the sake of flaming at all) could you please
give some usecases when such technique (described by Philip P.) backfires?
Building packages myself I am very interested in a ways to improve packaging.
For example: some packages I built contain enormous number of files (3K+) -
what do you do then? Obviously you can't enumerate all of them and it's hard
to track every single change even between upstream revisions (changed 10
filenames etc.)
If you feel more comfortable replying off-list please do so - either way I'm
interested in your opinion. thanks.

This is what I would do.

I would generate a file (lets say package.lst) with a listing of
files to include. Would do this manually, and then look at that
listing and see if everything is as it should be. For the sake of
documentation, I would put a comment inside the specfile with the
command used to generate that file.

That file would then be used with %files -f package.lst.

Then, when I upgrade the software, I would generate a second file listing,
and compare the 2 (diff -u, etc). If everything was fine, I would use the
new one.

Another option would be to include that listing directly on the
specfile, or just the files that need something different than
%defattr.

The idea is that you have control of what is there, and so you avoid
surprises.

Ok, everything will be fine, if you just use a glob, 99% of the time.
So if you are managing just 1 or 2 packages, and know them well,
that should not be a problem. However, if you are managing several
packages, this way can save you a lot of headaches trying to find out
why that package you built is not working as its last version was.

- --
Rodrigo Barbosa
"Quid quid Latine dictum sit, altum viditur"
"Be excellent to each other ..." - Bill & Ted (Wyld Stallyns)

[Philip Prindeville]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Dmitry S. Makovey

Post by Rodrigo Barbosa

Post by Philip Prindeville
%{_libexecdir}/mod_*.so

This is such a bad practice I almost cried from reading this e-mail.
NEVER do this. Ever.
If you reuse your specfile for a new version of the package, you
will completely loose (developer) control of which files are in
your package.
Also name all your files.

Out of pure curiosity (and for the sake of flaming at all) could you please
give some usecases when such technique (described by Philip P.) backfires?
Building packages myself I am very interested in a ways to improve packaging.
For example: some packages I built contain enormous number of files (3K+) -
what do you do then? Obviously you can't enumerate all of them and it's hard
to track every single change even between upstream revisions (changed 10
filenames etc.)
If you feel more comfortable replying off-list please do so - either way I'm
interested in your opinion. thanks.

This is what I would do.
I would generate a file (lets say package.lst) with a listing of
files to include. Would do this manually, and then look at that
listing and see if everything is as it should be. For the sake of
documentation, I would put a comment inside the specfile with the
command used to generate that file.
That file would then be used with %files -f package.lst.
Then, when I upgrade the software, I would generate a second file listing,
and compare the 2 (diff -u, etc). If everything was fine, I would use the
new one.
Another option would be to include that listing directly on the
specfile, or just the files that need something different than
%defattr.
The idea is that you have control of what is there, and so you avoid
surprises.
Ok, everything will be fine, if you just use a glob, 99% of the time.
So if you are managing just 1 or 2 packages, and know them well,
that should not be a problem. However, if you are managing several
packages, this way can save you a lot of headaches trying to find out
why that package you built is not working as its last version was.

And... what happens if the RPM generates different lists of files based
on the state of the machine it's built on?

I.e. what if it auto-detects support libraries (like openssl-devel) that
might unlock additional functionality?

What if compilation on different processors or different distros changes
the output target package? If you're using the same .spec file on 80
different machines/releases/processor families, it's hard to believe
that this isn't the case at least occasionally.

-Philip

[Bob Proulx]

Post by Philip Prindeville
And... what happens if the RPM generates different lists of files based
on the state of the machine it's built on?

You should at least know this information.

Post by Philip Prindeville
I.e. what if it auto-detects support libraries (like openssl-devel) that
might unlock additional functionality?
What if compilation on different processors or different distros changes
the output target package? If you're using the same .spec file on 80
different machines/releases/processor families, it's hard to believe
that this isn't the case at least occasionally.

This all sounds a little bit scary. One of the very useful things
about rpm is to get control of the build process to create
reproducable builds. Of course rpm is only part of the equation
because as you point out the rest of the environment also affects
things too.

I guess if you don't have control of the environment and don't care
then it does not really matter what you put in the rpm file. It will
probably be completely different each and every time. :-(

I really hate it when things like a 5.97-5.3 to 5.97-5.4 version
difference in packages contains massive changes in libraries and
dependencies. That basically tells me that there is no quality
control in the build process.

The careful builders want to know exactly what changed between one
package version and another and will try to make sure these are
intentional changes and are noted in the package changelogs.

Bob

[Michael Jennings]

On Wednesday, 19 September 2007, at 16:17:45 (-0600),

Post by Dmitry S. Makovey
Out of pure curiosity (and for the sake of flaming at all) could you
please give some usecases when such technique (described by Philip
P.) backfires? Building packages myself I am very interested in a
ways to improve packaging.

It's not really a question of use case. It's a risk/benefit
tradeoff. Packages that hardcode specific file lists are not as
portable (e.g., across distros) and require greater maintenance for
updates (as installed files are added or removed). However, doing it
that way does mean that you'll "know" (by virtue of build failures) if
any capabilities appear or disappear. I would submit that that's the
wrong way to "know" such things, but I'm sure others would disagree.

Post by Dmitry S. Makovey
For example: some packages I built contain enormous number of files
(3K+) - what do you do then? Obviously you can't enumerate all of
them and it's hard to track every single change even between
upstream revisions (changed 10 filenames etc.)

You most certainly can enumerate them all. There's a difference
between "can't" and "would be a pain in the ass to." :-)

But again, it's a tradeoff: if you need the fine-grain dictatorial
control (which I refered to as fascist) over what each package
provides, you must tolerate the pain of complete manual file list
enumeration. But if you believe, as I do, that packaging is
descriptive, not prescriptive, using globs is not only reasonable but
encouraged.

Michael