Getting 3rd party RPM's via an OS installer?

[Dan Stromberg - Datallegro]

Say you have an installer program for a bunch of RPM's.

You want to add some 3rd party RPM's to the collection
of RPM's with which this installer has been tested.

But some of these 3rd party RPM's aren't very careful
with their Requires dependencies, so if you toss those
RPM's into the collection, a high percentage of them
fail to install.

Is there some way of constructing a special RPM that
should always be installed at the end of the tested
RPM's, and change minimally your 3rd party RPM's to
depend on that single RPM? That is, without having
a huge list of Requires in that special RPM?

Yes, it'd be cleaner to go fix all the dependencies
in those 3rd party RPM's, but that should be fixed
upstream... We may be able to file bug reports
with each vendor about their broken dependencies,
but 1) some may not care and 2) I doubt many of them
would be able to respond "soon enough" for our need

Thanks!

[Bob Proulx]

Post by Dan Stromberg - Datallegro
But some of these 3rd party RPM's aren't very careful
with their Requires dependencies,

Don't you just hate that?

Post by Dan Stromberg - Datallegro
so if you toss those RPM's into the collection, a high percentage of
them fail to install.

Sigh. Yes. Been there. Bought the t-shirt.

Post by Dan Stromberg - Datallegro
Is there some way of constructing a special RPM that
should always be installed at the end of the tested
RPM's, and change minimally your 3rd party RPM's to
depend on that single RPM? That is, without having
a huge list of Requires in that special RPM?

You could create an rpm that depended not only on the 3rd party rpms
but also on the dependencies that you have deduced are required. Then
install your rpm and it should bring in all of the dependencies.

Bob

[Dan Stromberg - Datallegro]

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
But some of these 3rd party RPM's aren't very careful
with their Requires dependencies,

Don't you just hate that?

Yeah, it's a pain.

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
Is there some way of constructing a special RPM that
should always be installed at the end of the tested
RPM's, and change minimally your 3rd party RPM's to
depend on that single RPM? That is, without having
a huge list of Requires in that special RPM?

You could create an rpm that depended not only on the 3rd party rpms
but also on the dependencies that you have deduced are required. Then
install your rpm and it should bring in all of the dependencies.

Say the special RPM is called "knot".

Wouldn't you need to make your 3rd party RPM's require knot, and make knot
require anything the 3rd party RPM's require?

I'm no RPM expert, but it seems like if knot required the 3rd party RPM's,
it'd be installed after the 3rd party RPM's, and the 3rd party RPM's
remain at the mercy of the topological sort. Or am I missing something?

[Tony Earnshaw]

Post by Dan Stromberg - Datallegro

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
But some of these 3rd party RPM's aren't very careful
with their Requires dependencies,

Don't you just hate that?

Yeah, it's a pain.

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
Is there some way of constructing a special RPM that
should always be installed at the end of the tested
RPM's, and change minimally your 3rd party RPM's to
depend on that single RPM? That is, without having
a huge list of Requires in that special RPM?

You could create an rpm that depended not only on the 3rd party rpms
but also on the dependencies that you have deduced are required. Then
install your rpm and it should bring in all of the dependencies.

Say the special RPM is called "knot".
Wouldn't you need to make your 3rd party RPM's require knot, and make knot
require anything the 3rd party RPM's require?
I'm no RPM expert, but it seems like if knot required the 3rd party RPM's,
it'd be installed after the 3rd party RPM's, and the 3rd party RPM's
remain at the mercy of the topological sort. Or am I missing something?

Well, since I moved from up2date to FC6, and then CentOS5 and RHL5, I've
found that yum is pretty good at sorting out dependencies (unless one
chooses stupid repositories - in which case all hell breaks loose).

With correct repositories (yeah, I once "updated" my whole FC7
installation with FC8 stuff) I've found yum to be unparalleled.

Anybody's shout :)

--Tonni

[Dan Stromberg - Datallegro]

Post by Tony Earnshaw

Post by Dan Stromberg - Datallegro

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
But some of these 3rd party RPM's aren't very careful
with their Requires dependencies,

Don't you just hate that?

Yeah, it's a pain.

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
Is there some way of constructing a special RPM that
should always be installed at the end of the tested
RPM's, and change minimally your 3rd party RPM's to
depend on that single RPM? That is, without having
a huge list of Requires in that special RPM?

You could create an rpm that depended not only on the 3rd party rpms
but also on the dependencies that you have deduced are required. Then
install your rpm and it should bring in all of the dependencies.

Say the special RPM is called "knot".
Wouldn't you need to make your 3rd party RPM's require knot, and make knot
require anything the 3rd party RPM's require?
I'm no RPM expert, but it seems like if knot required the 3rd party RPM's,
it'd be installed after the 3rd party RPM's, and the 3rd party RPM's
remain at the mercy of the topological sort. Or am I missing something?

Well, since I moved from up2date to FC6, and then CentOS5 and RHL5, I've
found that yum is pretty good at sorting out dependencies (unless one
chooses stupid repositories - in which case all hell breaks loose).
With correct repositories (yeah, I once "updated" my whole FC7
installation with FC8 stuff) I've found yum to be unparalleled.
Anybody's shout :)

IINM, yum's (and similar tool's) ability to deal with dependencies is only
as good as the Requires data inside the RPM's - but I'm faced with some
RPM's that don't have good Requires data. These troublesome RPM's are
assuming that the OS is already 100% installed and so do not need to
Require individual parts of the OS, while with the installation method
we're looking at, that isn't true - so for example a driver RPM that uses
"which make" without depending on the "which" package, can run into
trouble in this context, while it wouldn't in most contexts.

[Bob Proulx]

Post by Dan Stromberg - Datallegro

Post by Bob Proulx
You could create an rpm that depended not only on the 3rd party rpms
but also on the dependencies that you have deduced are required. Then
install your rpm and it should bring in all of the dependencies.

Say the special RPM is called "knot".
Wouldn't you need to make your 3rd party RPM's require knot, and make knot
require anything the 3rd party RPM's require?

If you can modify your 3rd party rpms directly then wouldn't it be
simpler to correct the Requires: in them directly too? If you can
modify the 3rd party rpms then there is no need for an intermediate
meta package to hold the extra dependencies. But if you can't then
that is why I suggested this other way instead. I admit it puts the
cart before the horse but it probably won't matter in many cases and
if you have one of those cases then it won't matter for you either.

Post by Dan Stromberg - Datallegro
I'm no RPM expert, but it seems like if knot required the 3rd party RPM's,
it'd be installed after the 3rd party RPM's, and the 3rd party RPM's
remain at the mercy of the topological sort. Or am I missing something?

You are right but it only matters if the 3rd party rpm %post script
uses one of the dependencies in the %post script itself. If the 3rd
party rpm does not do anything interesting in the %post then you would
be fine. After the rpm install finishes all of the dependencies would
be in place and subsequent command use would have all of the
dependencies installed.

In summary you are correct about the install ordering but if the
dependences are not needed until after the installation then as a
workaround kludge for not being able to modify the 3rd party rpms then
I think it will work anyway. If you can modify the 3rd party rpms
then simply fixing the Requires: to be correct is best.

You can inspect the rpm %post scripts with this following. If there
are no dependencies there then this hack I suggested would work. Of
course if you do find dependencies there then there is almost no
alternative to correcting the original rpm files.

rpm -qp --scripts file.rpm

Bob

[Dan Stromberg - Datallegro]

Post by Bob Proulx

Post by Dan Stromberg - Datallegro

Post by Bob Proulx
You could create an rpm that depended not only on the 3rd party rpms
but also on the dependencies that you have deduced are required. Then
install your rpm and it should bring in all of the dependencies.

Say the special RPM is called "knot".
Wouldn't you need to make your 3rd party RPM's require knot, and make knot
require anything the 3rd party RPM's require?

If you can modify your 3rd party rpms directly then wouldn't it be
simpler to correct the Requires: in them directly too? If you can
modify the 3rd party rpms then there is no need for an intermediate
meta package to hold the extra dependencies. But if you can't then
that is why I suggested this other way instead. I admit it puts the
cart before the horse but it probably won't matter in many cases and
if you have one of those cases then it won't matter for you either.

We have a program that can modify a binary RPM. It's not redistributable,
and isn't supported. We got it from another company.

However, even with that program, I'm trying to think ahead and imaginging
an interative process of adding dependencies again and again, as we change
OS's or upgrade 3rd party RPM's.

It seems like it might be more maintainable to have a single rpm that
provides no actual files, that serves as a sort of "knot - bow"
distinction. So any flakey RPM's depend on the knot, and the knot RPM
gets all the missing dependencies. I'm thinking this might pay off a bit
like the way the PNM graphics format reduces format conversions from
O(c*N**2) to O(c*2*N), though granted the constant is probably small.

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
I'm no RPM expert, but it seems like if knot required the 3rd party RPM's,
it'd be installed after the 3rd party RPM's, and the 3rd party RPM's
remain at the mercy of the topological sort. Or am I missing something?

You are right but it only matters if the 3rd party rpm %post script
uses one of the dependencies in the %post script itself. If the 3rd
party rpm does not do anything interesting in the %post then you would
be fine. After the rpm install finishes all of the dependencies would
be in place and subsequent command use would have all of the
dependencies installed.

Agreed, but i have a question. Could %pre scripts make a difference? Or
are they all run before any RPM's are installed at all?

Post by Bob Proulx
In summary you are correct about the install ordering but if the
dependences are not needed until after the installation then as a
workaround kludge for not being able to modify the 3rd party rpms then I
think it will work anyway. If you can modify the 3rd party rpms then
simply fixing the Requires: to be correct is best.

I see.

Post by Bob Proulx
You can inspect the rpm %post scripts with this following. If there are
no dependencies there then this hack I suggested would work. Of course
if you do find dependencies there then there is almost no alternative to
correcting the original rpm files.
rpm -qp --scripts file.rpm

Ah, OK.

Thanks :)

[Valery Reznic]

--- Dan Stromberg - Datallegro

On Tue, 21 Aug 2007 12:05:06 -0600, Bob Proulx

Post by Bob Proulx

Post by Dan Stromberg - Datallegro

Post by Bob Proulx
You could create an rpm that depended not only

on the 3rd party rpms

Post by Bob Proulx

Post by Dan Stromberg - Datallegro

Post by Bob Proulx
but also on the dependencies that you have

deduced are required. Then

Post by Bob Proulx

Post by Dan Stromberg - Datallegro

Post by Bob Proulx
install your rpm and it should bring in all of

the dependencies.

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
Say the special RPM is called "knot".
Wouldn't you need to make your 3rd party RPM's

require knot, and make knot

Post by Bob Proulx

Post by Dan Stromberg - Datallegro
require anything the 3rd party RPM's require?

If you can modify your 3rd party rpms directly

then wouldn't it be

Post by Bob Proulx
simpler to correct the Requires: in them directly

too? If you can

Post by Bob Proulx
modify the 3rd party rpms then there is no need

for an intermediate

Post by Bob Proulx
meta package to hold the extra dependencies. But

if you can't then

Post by Bob Proulx
that is why I suggested this other way instead. I

admit it puts the

Post by Bob Proulx
cart before the horse but it probably won't matter

in many cases and

Post by Bob Proulx
if you have one of those cases then it won't

matter for you either.
We have a program that can modify a binary RPM. It's
not redistributable,
and isn't supported. We got it from another
company.

You can try rpmrebuild (http://rpmrebuild.sf.net)
And if your program has (useful :) features, that
rpmrebuild not - could you tell me ?

However, even with that program, I'm trying to think
ahead and imaginging
an interative process of adding dependencies again
and again, as we change
OS's or upgrade 3rd party RPM's.

rpmrebuild allows batch processing.

Valery.




____________________________________________________________________________________
Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.
http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow

[Leland Ray]

Determining what the precise requirements are is a
common problem.

One thing that would make it easier is to make
requires more testable. It would be useful to have a
flag somewhere that suggests to the installation
method to move the package as close to the front of
installation order as possible.

Another testing problem is that when an install fails
to find a requirement, whether that error halts
installation depends on the installer. So anaconda
will permit installation of components where not all
requirements are present, but other installation
methods will not.

This is going to seem anathema to many, but because I
don't have control over third party packaging, I'd
like to be able to add requires to a package without
rebuilding it. My dream is to be able to edit
requires-additional.xml in a yum repository, and have
anaconda, if in a kickstart, or yum, or any other
package, honor the extra requires as if they were part
of the rpm.

Hmmm, perhaps I should wish for this on the yum list.



____________________________________________________________________________________
Got a little couch potato?
Check out fun summer activities for kids.
http://search.yahoo.com/search?fr=oni_on_mail&p=summer+activities+for+kids&cs=bz

[Bob Proulx]

Post by Leland Ray
One thing that would make it easier is to make
requires more testable. It would be useful to have a
flag somewhere that suggests to the installation
method to move the package as close to the front of
installation order as possible.

The problem is what do you do when there are multiple packages hinting
to be "as early as possible"? What should the ordering be there? I
don't think imprecise hinting is the right way to go. Better to have
the actual requirements and then be able to sort based upon those.

Post by Leland Ray
Another testing problem is that when an install fails
to find a requirement, whether that error halts
installation depends on the installer. So anaconda
will permit installation of components where not all
requirements are present, but other installation
methods will not.

I think this is because of an attempt to work around circular
dependencies. I really wish circular dependencies were simply
disallowed. It would solve many problems.

Post by Leland Ray
This is going to seem anathema to many, but because I
don't have control over third party packaging, I'd
like to be able to add requires to a package without
rebuilding it.

It is mostly possible to create a new rpm based upon an existing rpm
simply by repackaging the binary rpm and without recompiling. There
are some issues trying to do this in the general case but most
particular examples can be recreated.

Post by Leland Ray
My dream is to be able to edit requires-additional.xml in a yum
repository, and have anaconda, if in a kickstart, or yum, or any
other package, honor the extra requires as if they were part of the
rpm.
Hmmm, perhaps I should wish for this on the yum list.

It would be more palatable if that fed directly into a bug database
such that whenever it was used it was clearly indicated as a
workaround for bugs in the packaging. I would fear that something
like this would become accepted as a normal way to do things and not
as an exceptional condition to work around bugs.

Bob

[Dan Stromberg - Datallegro]

Post by Bob Proulx

Post by Leland Ray
One thing that would make it easier is to make
requires more testable. It would be useful to have a
flag somewhere that suggests to the installation
method to move the package as close to the front of
installation order as possible.

The problem is what do you do when there are multiple packages hinting
to be "as early as possible"? What should the ordering be there? I
don't think imprecise hinting is the right way to go. Better to have
the actual requirements and then be able to sort based upon those.

One simple fix is to have "knots" and "bows". Each loop of a bow is a
topological sort; each knot is an inflection point that cordons off one
tsort from another.

So the OS would go into bow 0, and most of the time, all the other RPM's
would go into bow 1. Or you could use them bows like BASIC line numbers -
starting with the OS in bow 1000, allowing room for things to come in
front and behind.

Then as you compute your tsorts, you just catenate them together in bow
order to get your ordered list of rpm's to install.

Post by Bob Proulx

Post by Leland Ray
Another testing problem is that when an install fails to find a
requirement, whether that error halts installation depends on the
installer. So anaconda will permit installation of components where not
all requirements are present, but other installation methods will not.

I think this is because of an attempt to work around circular
dependencies. I really wish circular dependencies were simply
disallowed. It would solve many problems.

One thing that might help is if installers had an "install in random order
(within tsort constraints).

Post by Bob Proulx

Post by Leland Ray
This is going to seem anathema to many, but because I don't have
control over third party packaging, I'd like to be able to add requires
to a package without rebuilding it.

It is mostly possible to create a new rpm based upon an existing rpm
simply by repackaging the binary rpm and without recompiling. There are
some issues trying to do this in the general case but most particular
examples can be recreated.

I've seen a program that allows you to edit binary RPM's, but it's
copyrighted with a restrictive license at this time.

Post by Bob Proulx

Post by Leland Ray
My dream is to be able to edit requires-additional.xml in a yum
repository, and have anaconda, if in a kickstart, or yum, or any other
package, honor the extra requires as if they were part of the rpm.
Hmmm, perhaps I should wish for this on the yum list.

It would be more palatable if that fed directly into a bug database such
that whenever it was used it was clearly indicated as a workaround for
bugs in the packaging. I would fear that something like this would
become accepted as a normal way to do things and not as an exceptional
condition to work around bugs.

As long as it's a pain in the rear (and IMO, if it doesn't "just work",
it is a pain), I think people will avoid it.

But adding an XML file with additional requires dependencies to an
installer, probably wouldn't be any easier than modifying rpm (the program
that installs rpm packages's) to allow adding and removing requires
dependencies.

[Jeff Johnson]

Post by Dan Stromberg - Datallegro
One simple fix is to have "knots" and "bows". Each loop of a bow is a
topological sort; each knot is an inflection point that cordons off one
tsort from another.
So the OS would go into bow 0, and most of the time, all the other RPM's
would go into bow 1. Or you could use them bows like BASIC line numbers -
starting with the OS in bow 1000, allowing room for things to come in
front and behind.
Then as you compute your tsorts, you just catenate them together in bow
order to get your ordered list of rpm's to install.

This works for ordering, not for dependency closure.

73 de Jeff

[Dan Stromberg - Datallegro]

Post by Jeff Johnson

Post by Dan Stromberg - Datallegro
One simple fix is to have "knots" and "bows". Each loop of a bow is a
topological sort; each knot is an inflection point that cordons off one
tsort from another.
So the OS would go into bow 0, and most of the time, all the other RPM's
would go into bow 1. Or you could use them bows like BASIC line numbers -
starting with the OS in bow 1000, allowing room for things to come in
front and behind.
Then as you compute your tsorts, you just catenate them together in bow
order to get your ordered list of rpm's to install.

This works for ordering, not for dependency closure.

It seems to me like it should work fine. What am I missing?

Granted, you could have dependencies that contradict your bow
constraints... But then you could have an rpm option that says to ignore
dependencies between bows.

[Jeff Johnson]

Post by Dan Stromberg - Datallegro

Post by Jeff Johnson
This works for ordering, not for dependency closure.

It seems to me like it should work fine. What am I missing?

For partial ordering through dependencies, P -> Q (as in order Q
before P) relations are
meaningless and harmless when Q doesn't exist.

For dependency closure, P -> Q (as in package P has Requires: Q) is
full stop failure.

There are many name space issues, when Q and Q' are functionally
equivalent
but not identically named, that dependency closure is more
susceptible than
ordering to failures.

Post by Dan Stromberg - Datallegro
Granted, you could have dependencies that contradict your bow
constraints... But then you could have an rpm option that says to ignore
dependencies between bows.

One could ignore dependencies between bows, but then bow 0 may not be
populated with something necessary for a bow 1 package.

73 de Jeff

[Leland Ray]

Post by Bob Proulx

Post by Leland Ray
One thing that would make it easier is to make

requires more testable.

Post by Bob Proulx

Post by Leland Ray
It would be useful to have a flag somewhere that

suggests to the

Post by Bob Proulx

Post by Leland Ray
installation method to move the package as close to

the front of

Post by Bob Proulx

Post by Leland Ray
installation order as possible.

The problem is what do you do when there are multiple
packages hinting to be "as early as possible"? What
should the ordering be there? I don't think

imprecise >hinting is the right way to go. Better to
have the >actual requirements and then be able to sort
based upon

Post by Bob Proulx
those.

Ahh but the point is, right now requirements aren't
easily testable. I would like people developing rpms
to see missing requirements during their normal
testing.

So what I am suggesting is to have a way to request an
early install. Thus an rpm that didn't have all
requirements listed would be likely to fail during
development.

Where that flag goes is a really good question. If it
is part of the rpm, its too likely that people will
mistakenly ship software with the flag set. So I'm now
thinking that this isn't an rpm issue, but should be
part of the repository, once again.

Post by Bob Proulx

Post by Leland Ray
My dream is to be able to edit

requires-additional.xml in a yum

Post by Bob Proulx

Post by Leland Ray
repository, and have anaconda, if in a kickstart,

or yum, or any other

Post by Bob Proulx

Post by Leland Ray
package, honor the extra requires as if they were

part of the rpm.

Post by Bob Proulx

Post by Leland Ray
Hmmm, perhaps I should wish for this on the yum

list.

Post by Bob Proulx
It would be more palatable if that fed directly into

a >bug database such that whenever it was used it was

Post by Bob Proulx
clearly indicated as a workaround for bugs in the
packaging. I would fear that something like this
would become accepted as a normal way to do things

and >not as an exceptional condition to work around
bugs.

I agree, it would be better if people actually fixed
their bugs.



____________________________________________________________________________________
Building a website is a piece of cake. Yahoo! Small Business gives you all the tools to get online.
http://smallbusiness.yahoo.com/webhosting

[Bob Proulx]

Post by Leland Ray
Ahh but the point is, right now requirements aren't
easily testable. I would like people developing rpms
to see missing requirements during their normal
testing.

Install into a minimum chroot. Anything that is missing will be
noticed in that environment.

Post by Leland Ray
So what I am suggesting is to have a way to request an
early install. Thus an rpm that didn't have all
requirements listed would be likely to fail during
development.

But during development the developer will very likely have all of the
dependencies installed already. The only way to catch this is to
install into a system that does not have the dependencies. This is
often done with chroots.

Post by Leland Ray
I agree, it would be better if people actually fixed
their bugs.

Full agreement! :-)

Bob