... binary RPM question ...

[Novick, Randy]

Hi,

This is my first post to the list, though I've been lurking for a few
days.

I have been reading Maximun RPM and a host of other howtos and I've got
a .spec file in progress, but I'm confused about how to do something
that appears to not fit so neatly into the documentation I've read.

The challenge is this:

We have our own build process that creates all the necessary object
files on each of our UNIX platforms (there are three, and AIX and Linux
are the two I'm interested in delivering via RPM). We cannot ship our
sources to our customers, yet we would like to leverage the RPM database
to track and manage what files get installed at the customer site.

We have a packaging process that follows compilation which plucks all of
the objects and library dependencies into a "runtime" directory that
represents what the customer will have installed. This "runtime"
directory is tarred up and gzipped and this is what we deliver to the
customer as a release, also shipping a set of scripts alongside that
handle the gunzipping and untarring of the deliverable, the creation of
logdirs, the setup of config files and the tweaking of system settings
necessary for getting things to run.

What I want to do is to create an RPM that contains what I've compiled
in the form of the gzipped tar I create, and also encapsulates the
scripting necessary to untar, create dirs, setup configs and tweak
settings in the .spec file.

I'm stymied, though, as to how to use the rpmbuild dirs to achieve this.
Which ones should I not use, if I'm not having the customer compile the
code as a part of RPM installation? What steps in the .spec file can I
skip? Do I get any advantage out of not pre-tarring-and-gzipping my
build products? Which steps in the .spec file must I certainly use to
contain my scripting?

I've had a lot of experience with solaris packaging, so I understand
preinstall, install, postinstall ... and preun, un and postun. Rpmbuild
has got a lot more complexity, though, and appears to be more suited to
open source deliveries than the kind of closed-source installs I'm
trying to achieve.

I'd appreciate some advice, and some help getting started.
Thanks,
-- R





Randy Novick

Software Configuration Management

McKesson Provider Technologies

***@mckesson.com

[phone]



Confidentiality Notice: This email message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
confidential and privileged information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply email and destroy all
copies of the original message.

[Leland Ray]

Yes, what you are asking to do is quite possible. One
change I suggest is to use an unpacked runtime
directory for packaging. This would allow rpm to keep
track of each installed file in your system (so rpm
-qf would work).

There are two choices to how to call rpmbuild in this
case. First method is if you wanted to use a
tar/gzipped runtime direcotry as input to rpm
packaging. Second method is if you want to just
produce rpms at the end of your build process.

For method 1:

Your spec would skip the %pre and %build step, and
would have no Source: tag.

The %install scriptlet would create $RPM_BUILD_ROOT,
and untar / unzip your runtime tree.

For %files, you can probably use just a wildcard,
assuming your build system doesn't include extraneous
files in the runtime directory.

In %clean, you would delete $RPM_BUILD_ROOT.

---

For method 2:

The build system would call rpmbuild prior to tar/gzip
of the runtime directory. You would redefine macros on
the command line to rpmbuild so that to rpm, the build
root is the same as the base of the runtime directory.

You may also need to define several other macros, but
knowing about the one will get you started.

For this case, the .spec file would not need %prep,
%build, or %install. It would just need %files and the
pre/post/preun/postun scriptlets as required. It would
also omit Source:



____________________________________________________________________________________
Park yourself in front of a world of choices in alternative vehicles. Visit the Yahoo! Auto Green Center.
http://autos.yahoo.com/green_center/

[Bob Proulx]

Leland Ray wrote:
> Yes, what you are asking to do is quite possible. One
> [...good suggestions...]
> For %files, you can probably use just a wildcard,
> assuming your build system doesn't include extraneous
> files in the runtime directory.

Assuming that the files are placed in a top level directory the simply
listing the top level directory in the %files section would be good.
Then the directory would also be owned and -qf would work there. In
this case no wildcards would be used.

Bob

[Jos Vos]

On Wed, Jun 20, 2007 at 04:53:17PM -0600, Bob Proulx wrote:

> Assuming that the files are placed in a top level directory the simply
> listing the top level directory in the %files section would be good.
> Then the directory would also be owned and -qf would work there. In
> this case no wildcards would be used.

But *only* if that toplevel directory itself is package-specific.
Be sure to not include any system directory.

--
-- Jos Vos <***@xos.nl>
-- X/OS Experts in Open Systems BV | Phone: [phone]
-- Amsterdam, The Netherlands | Fax: [phone]

[Jos Vos]

On Wed, Jun 20, 2007 at 02:33:51PM -0700, Leland Ray wrote:

> There are two choices to how to call rpmbuild in this
> case. First method is if you wanted to use a
> tar/gzipped runtime direcotry as input to rpm
> packaging. Second method is if you want to just
> produce rpms at the end of your build process.
>
> For method 1:
>
> Your spec would skip the %pre and %build step, and
> would have no Source: tag.

If you don't have a Source: tag, where do you get your files from?
Out of the blue? From "what's there at that moment"?

This is against common RPM rules, even those that apply to packaging
binary software.

> The %install scriptlet would create $RPM_BUILD_ROOT,
> and untar / unzip your runtime tree.

Right, and your "runtime tree" should be included as "source".

> For %files, you can probably use just a wildcard,
> assuming your build system doesn't include extraneous
> files in the runtime directory.

This depends. *Only* of that extracted tree contains, say,
/usr/lib/myproduct/... and nothing else, you can include
just /usr/lib/myproduct. But never include a system
directory (like /usr, /usr/lib, etc.) in your binary RPM,
so wildcards like /usr/* or /* should never be used.

> The build system would call rpmbuild prior to tar/gzip
> of the runtime directory. You would redefine macros on
> the command line to rpmbuild so that to rpm, the build
> root is the same as the base of the runtime directory.
>
> You may also need to define several other macros, but
> knowing about the one will get you started.

Also against the way RPM is intended to be used.

The proper way, IMHO, is using the tar'ed binary tree as source
file.

...
Source: myproduct.tar.gz
...

%prep
# create dir before untarring
%setup -c

%build
# probably empty

%install
mkdir -p %{buildroot}
# assuming you tar has a directory tree from the root
cp -a * %{buildroot}/

%clean
rm -rf %{buildroot}

%files
%defattr(-,root,root)
# whatever is appropriate
/usr/bin/*
/usr/lib/*


The shortcut could be to not extract the source in %prep, but to
untar that manually in %install:

cd %{buildroot}
tar -xzf %{SOURCE0}


But in my experience, having packaged many third-party binary stuff,
you often want to tweak some things (maybe not needed in your case).


Have fun,

--
-- Jos Vos <***@xos.nl>
-- X/OS Experts in Open Systems BV | Phone: [phone]
-- Amsterdam, The Netherlands | Fax: [phone]

[Bob Proulx]

Jos Vos wrote:
> If you don't have a Source: tag, where do you get your files from?
> Out of the blue? From "what's there at that moment"?

In this case of people trying to use rpm for closed sourced projects
then usually the files will be installed by methods outside of rpm.
So from your perspective, yes, they will appear "out of the blue".
Of course this is frowned upon by free software advocates because it
prevents the source from being distributed. But as far as producing a
workable rpm file this is okay from a technical perspective.

> This is against common RPM rules, even those that apply to packaging
> binary software.

It is against most software rules/guidelines for distributing packages
with a distribution (e.g. Fedora) but not against rpmbuild rules. The
rpmbuild program allows this fine. The rpm process is okay with this.
It is a philosophical issue and not a technical one.

> The proper way, IMHO, is using the tar'ed binary tree as source
> file.

Of course doing it this way is fine. But it is not required. Among
other things bundling a binary blob in like that allows a .src.rpm
file to be produced that could rebuild the binary .rpm file. But
because the source is not really source the .src.rpm file is not
really useful. It could not be used to port the code from 32-bit to
64-bit for example. Therefore IMNHO in this type of case it does not
really add anything over just having the files appear out of the blue.

Bob

[Jos Vos]

On Fri, Jun 22, 2007 at 03:10:29PM -0600, Bob Proulx wrote:

> In this case of people trying to use rpm for closed sourced projects
> then usually the files will be installed by methods outside of rpm.

The only difference between open and closed source here is that the
"source" files (rpm speak) are sources in the first case and binaries
in the latter case.

> So from your perspective, yes, they will appear "out of the blue".

Out of the blue == outside anything specified by (rpm speak) sources
and patches. There is usually no good reason to create packages that
use files out of the blue.

> It is against most software rules/guidelines for distributing packages
> with a distribution (e.g. Fedora) but not against rpmbuild rules. The
> rpmbuild program allows this fine. The rpm process is okay with this.
> It is a philosophical issue and not a technical one.

You can abuse any software, yes, also rpmbuild, so in that sense it's
not a technical issue.

> Of course doing it this way is fine. But it is not required. Among
> other things bundling a binary blob in like that allows a .src.rpm
> file to be produced that could rebuild the binary .rpm file. But
> because the source is not really source the .src.rpm file is not
> really useful. It could not be used to port the code from 32-bit to
> 64-bit for example. Therefore IMNHO in this type of case it does not
> really add anything over just having the files appear out of the blue.

I totally disagree, sorry.

--
-- Jos Vos <***@xos.nl>
-- X/OS Experts in Open Systems BV | Phone: [phone]
-- Amsterdam, The Netherlands | Fax: [phone]

[Michael Jennings]

On Friday, 22 June 2007, at 15:10:29 (-0600),
Bob Proulx wrote:

> In this case of people trying to use rpm for closed sourced projects
> then usually the files will be installed by methods outside of rpm.
> So from your perspective, yes, they will appear "out of the blue".
> Of course this is frowned upon by free software advocates because it
> prevents the source from being distributed. But as far as producing
> a workable rpm file this is okay from a technical perspective.

Only if you consider "seat of the pants" builds technically
acceptable.

> It is against most software rules/guidelines for distributing
> packages with a distribution (e.g. Fedora) but not against rpmbuild
> rules. The rpmbuild program allows this fine. The rpm process is
> okay with this. It is a philosophical issue and not a technical
> one.

Ditto above.

> Of course doing it this way is fine. But it is not required.

For those who feel reproducable, consistent builds are not desireable
or technically advantageous, I'm sure this is correct.

> Among other things bundling a binary blob in like that allows a
> .src.rpm file to be produced that could rebuild the binary .rpm
> file. But because the source is not really source the .src.rpm file
> is not really useful.

I'm starting to feel like a broken record here.

> It could not be used to port the code from 32-bit to 64-bit for
> example. Therefore IMNHO in this type of case it does not really
> add anything over just having the files appear out of the blue.

Couldn't disagree more.

Michael

--
Michael Jennings (a.k.a. KainX) http://www.kainx.org/ <***@kainx.org>
n + 1, Inc., http://www.nplus1.net/ Author, Eterm (www.eterm.org)
-----------------------------------------------------------------------
"Did you really have to die for me? All I am for all You are because
What I need and what I believe are worlds apart."
-- Jars of Clay, "Worlds Apart"

[Hiren Patel]

On Fri, 2007-06-22 at 15:10 -0600, Bob Proulx wrote:
> Jos Vos wrote:
> > If you don't have a Source: tag, where do you get your files from?
> > Out of the blue? From "what's there at that moment"?
>
> In this case of people trying to use rpm for closed sourced projects
> then usually the files will be installed by methods outside of rpm.
> So from your perspective, yes, they will appear "out of the blue".
> Of course this is frowned upon by free software advocates because it
> prevents the source from being distributed. But as far as producing a
> workable rpm file this is okay from a technical perspective.

if you have to deviate from best practices, why not deviate only where
completely necessary? if something can be done with or without
deviating, why not choose not to deviate?

>
> > This is against common RPM rules, even those that apply to packaging
> > binary software.
>
> It is against most software rules/guidelines for distributing packages
> with a distribution (e.g. Fedora) but not against rpmbuild rules. The
> rpmbuild program allows this fine. The rpm process is okay with this.
> It is a philosophical issue and not a technical one.
>

IMHO, most issues are philosophical, technicality must complement
philosophy.

> > The proper way, IMHO, is using the tar'ed binary tree as source
> > file.
>
> Of course doing it this way is fine. But it is not required. Among
> other things bundling a binary blob in like that allows a .src.rpm
> file to be produced that could rebuild the binary .rpm file. But
> because the source is not really source the .src.rpm file is not
> really useful. It could not be used to port the code from 32-bit to
> 64-bit for example. Therefore IMNHO in this type of case it does not
> really add anything over just having the files appear out of the blue.
>

i disagree.
though i still think the source rpm should have the actual sources, to
make it useful obviously. i also think the build process should be done
via rpm, but if that is not possible, at least have the source rpm have
the actual source code tarball as SOURCE's, so that it is not an
unusable blob.

> Bob
>
> _______________________________________________
> Rpm-list mailing list
> Rpm-***@redhat.com
> https://www.redhat.com/mailman/listinfo/rpm-list
--

Hiren Patel | Ops Specialist | ISS Infrastructure | Telkom
E-Mail: ***@telkom.co.za Office: [phone] | Fax: [phone]
3299 | Cell: [phone]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This e-mail and its contents are subject to the Telkom SA Limited
e-mail legal notice available at
http://www.telkom.co.za/TelkomEMailLegalNotice.PDF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Bob Proulx]

Hiren Patel wrote:
> Bob Proulx wrote:
> > In this case of people trying to use rpm for closed sourced projects
> > then usually the files will be installed by methods outside of rpm.
> > So from your perspective, yes, they will appear "out of the blue".
> > Of course this is frowned upon by free software advocates because it
> > prevents the source from being distributed. But as far as producing a
> > workable rpm file this is okay from a technical perspective.
>
> if you have to deviate from best practices, why not deviate only where
> completely necessary? if something can be done with or without
> deviating, why not choose not to deviate?

Hiren I appreciated your question because it was thoughtful.

What you are seeing is a collision of world views. For the free
software advocate anything that does not provide the source in an
easily accesable form is a violation of free software best practice.
I agree completely. Not using rpm to build and not producing a
src.rpm file is a violation of good free software best practice.

However for the closed source proprietary camp providing source isn't
going to happen. The best practice in the closed source camp is never
making the source available. These two views, free software and
closed source software, are diametrically opposed philosophical views
that will never converge. As such they have different best practices
and for close source using rpm this way is not violating their best
practices even if it is violating those of a free software advocate.

I don't like closed source proprietary software but the rpm license
does not prevent using rpm to package it and manage the installation
of it. In fact if it did then rpm itself could not be considered free
software. As such rpm is available for use managing closed source
proprietary software. The license allows it. There are no technical
barriers to it. There are many advantages to using rpm to manage
software installations. Use of rpm packages is requested of software
vendors by their customers. It is only natural that closed source
proprietary vendors will use rpm packages for their closed source
software.

Using rpm for closed source binary packages will annoy those who
believe that any process other than the free software process is a bad
deviation. It annoys me. But this cannot be restricted without also
restricting rpm such that it is no longer free software. That would
be far worse.

> IMHO, most issues are philosophical, technicality must complement
> philosophy.

No need to be humble about that opinion because you are exactly
correct about it being a philosophical issue.

> > > The proper way, IMHO, is using the tar'ed binary tree as source
> > > file.
> >
> > Of course doing it this way is fine. But it is not required. Among
> > other things bundling a binary blob in like that allows a .src.rpm
> > file to be produced that could rebuild the binary .rpm file. But
> > because the source is not really source the .src.rpm file is not
> > really useful. It could not be used to port the code from 32-bit to
> > 64-bit for example. Therefore IMNHO in this type of case it does not
> > really add anything over just having the files appear out of the blue.
> >
>
> i disagree.

What perspective do you assume when you disagree with the above? The
admin consumer of the rpm when you install it? Or the developer
builder of the rpm when you create it?

It is always possible to tar up the binary files and use that as a
source file. But if that is all that it is doing I don't see the
utility of it. For example (a real example from my experience) let's
say that the compressed tar file of the installed files is 5G. It
takes a large amount of disk to store both the original files and the
tar.gz and the src.rpm file plus the produced binary rpm file. Using
an extra 10G to store the tar.gz and the src.rpm on top of the already
required 10G for the original native files plus binary rpm is a lot of
overhead.

Also it can take a long time even on a fast system to move that much
data around. Compressing and uncompressing that much data repeatedly
is cpu intensive. Considering the fact that this does not include the
source and that no one will ever rebuild from the src.rpm file it is
not worth the effort to maintain it. It takes so much time and disk
space that I eventually gave up fighting for it. This is just one
example.

> though i still think the source rpm should have the actual sources, to
> make it useful obviously. i also think the build process should be done
> via rpm, but if that is not possible, at least have the source rpm have
> the actual source code tarball as SOURCE's, so that it is not an
> unusable blob.

It is always possible to build within the rpm. However it is often
not practical to spend the effort to convert and clean up some horrid
private-sector build mess to the point that it can be built in rpm.
By not practical I mean because the boss won't pay for it. Most
software projects that I have seen in the corporate world were grown
in situ using ad-hoc development and need a lot of cleaning but don't
get what they need. Some of the company processes that I have seen
are absolutely terrible. Cleaning up a closed source proprietary
project in the private sector can be a very problematic task.

Remember that closed source projects in the private sector are
rarely as high quality as publicly developed free software projects.
They don't get the same level of review given to public projects.
They don't have access to as much talent as public projects.

Bob

[Hiren Patel]

hi,

thanks for the reply Bob.

i respect the opinions of others and will not continue to comment on
this thread, at the end of the day, one should do as one feels is
appropriate, others are welcome to advise and comment.

On Mon, 2007-06-25 at 22:23 -0600, Bob Proulx wrote:
> Hiren Patel wrote:
> > Bob Proulx wrote:
> > > In this case of people trying to use rpm for closed sourced projects
> > > then usually the files will be installed by methods outside of rpm.
> > > So from your perspective, yes, they will appear "out of the blue".
> > > Of course this is frowned upon by free software advocates because it
> > > prevents the source from being distributed. But as far as producing a
> > > workable rpm file this is okay from a technical perspective.
> >
> > if you have to deviate from best practices, why not deviate only where
> > completely necessary? if something can be done with or without
> > deviating, why not choose not to deviate?
>
> Hiren I appreciated your question because it was thoughtful.
>
> What you are seeing is a collision of world views. For the free
> software advocate anything that does not provide the source in an
> easily accesable form is a violation of free software best practice.
> I agree completely. Not using rpm to build and not producing a
> src.rpm file is a violation of good free software best practice.
>
> However for the closed source proprietary camp providing source isn't
> going to happen. The best practice in the closed source camp is never
> making the source available. These two views, free software and
> closed source software, are diametrically opposed philosophical views
> that will never converge. As such they have different best practices
> and for close source using rpm this way is not violating their best
> practices even if it is violating those of a free software advocate.
>
> I don't like closed source proprietary software but the rpm license
> does not prevent using rpm to package it and manage the installation
> of it. In fact if it did then rpm itself could not be considered free
> software. As such rpm is available for use managing closed source
> proprietary software. The license allows it. There are no technical
> barriers to it. There are many advantages to using rpm to manage
> software installations. Use of rpm packages is requested of software
> vendors by their customers. It is only natural that closed source
> proprietary vendors will use rpm packages for their closed source
> software.
>
> Using rpm for closed source binary packages will annoy those who
> believe that any process other than the free software process is a bad
> deviation. It annoys me. But this cannot be restricted without also
> restricting rpm such that it is no longer free software. That would
> be far worse.
>
> > IMHO, most issues are philosophical, technicality must complement
> > philosophy.
>
> No need to be humble about that opinion because you are exactly
> correct about it being a philosophical issue.
>
> > > > The proper way, IMHO, is using the tar'ed binary tree as source
> > > > file.
> > >
> > > Of course doing it this way is fine. But it is not required. Among
> > > other things bundling a binary blob in like that allows a .src.rpm
> > > file to be produced that could rebuild the binary .rpm file. But
> > > because the source is not really source the .src.rpm file is not
> > > really useful. It could not be used to port the code from 32-bit to
> > > 64-bit for example. Therefore IMNHO in this type of case it does not
> > > really add anything over just having the files appear out of the blue.
> > >
> >
> > i disagree.
>
> What perspective do you assume when you disagree with the above? The
> admin consumer of the rpm when you install it? Or the developer
> builder of the rpm when you create it?
>
> It is always possible to tar up the binary files and use that as a
> source file. But if that is all that it is doing I don't see the
> utility of it. For example (a real example from my experience) let's
> say that the compressed tar file of the installed files is 5G. It
> takes a large amount of disk to store both the original files and the
> tar.gz and the src.rpm file plus the produced binary rpm file. Using
> an extra 10G to store the tar.gz and the src.rpm on top of the already
> required 10G for the original native files plus binary rpm is a lot of
> overhead.
>
> Also it can take a long time even on a fast system to move that much
> data around. Compressing and uncompressing that much data repeatedly
> is cpu intensive. Considering the fact that this does not include the
> source and that no one will ever rebuild from the src.rpm file it is
> not worth the effort to maintain it. It takes so much time and disk
> space that I eventually gave up fighting for it. This is just one
> example.
>
> > though i still think the source rpm should have the actual sources, to
> > make it useful obviously. i also think the build process should be done
> > via rpm, but if that is not possible, at least have the source rpm have
> > the actual source code tarball as SOURCE's, so that it is not an
> > unusable blob.
>
> It is always possible to build within the rpm. However it is often
> not practical to spend the effort to convert and clean up some horrid
> private-sector build mess to the point that it can be built in rpm.
> By not practical I mean because the boss won't pay for it. Most
> software projects that I have seen in the corporate world were grown
> in situ using ad-hoc development and need a lot of cleaning but don't
> get what they need. Some of the company processes that I have seen
> are absolutely terrible. Cleaning up a closed source proprietary
> project in the private sector can be a very problematic task.
>
> Remember that closed source projects in the private sector are
> rarely as high quality as publicly developed free software projects.
> They don't get the same level of review given to public projects.
> They don't have access to as much talent as public projects.
>
> Bob
>
> _______________________________________________
> Rpm-list mailing list
> Rpm-***@redhat.com
> https://www.redhat.com/mailman/listinfo/rpm-list
--

Hiren Patel | Ops Specialist | ISS Infrastructure | Telkom
E-Mail: ***@telkom.co.za Office: [phone] | Fax: [phone]
3299 | Cell: [phone]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This e-mail and its contents are subject to the Telkom SA Limited
e-mail legal notice available at
http://www.telkom.co.za/TelkomEMailLegalNotice.PDF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Jos Vos]

On Mon, Jun 25, 2007 at 10:23:52PM -0600, Bob Proulx wrote:

> It is always possible to tar up the binary files and use that as a
> source file. But if that is all that it is doing I don't see the
> utility of it. For example (a real example from my experience) let's
> say that the compressed tar file of the installed files is 5G. It
> takes a large amount of disk to store both the original files and the
> tar.gz and the src.rpm file plus the produced binary rpm file. Using
> an extra 10G to store the tar.gz and the src.rpm on top of the already
> required 10G for the original native files plus binary rpm is a lot of
> overhead.

OK, one other attempt then to explain it and then I really stop ;-):

Normally, when I package binary (proprietary) software, I start with
something I get from the vendor. Often a tar file with binaries and
an install script, but some vendors produce more horrible things.

When I then get a new version from the vendor, I can compare the
tarball with the previous version (what files are in it, has the
install script been changed, etc.) because I have the old, original
vendor's tarball included in my src.rpm.

If I would have installed the sofware with the install script and
made the rpm without "Source" files, I wouldn't be able to compare
what the vendor provides and I might have big trouble in "porting"
my rpm to the new version. This is just one example. I can think
of many more.

I agree that in the case the rpm build is a step in a larger build
process (that is, you're yourself the original vendor of the
proprietary software) things are a bit different. In that case your
own build process is the controlled way of generating the rpm and
you don't need to regenerate the rpm with stuff from someone else.
I can imagine you then make other decisions.

But in general the (IMHO) GOLDEN RULE (I don't know if this is ever
written down somewhere and maybe someone can rephrase it) is:

The files in the resulting binary rpm should *all* be generated from
*only* the Source and Patch files specified in the spec file.

Note that these "Source" files do not need to be sources, they can
also be (tarballs with) binaries.

And generate means: compile (in case of real sources), but it can
also be copy (in case of binary rpm's).

Note that this rule is never completely true: the result is also
dependent on compiler versions, versions of used include files and
libraries, etc., so (unfortunately) it's not perfect. But at least
it is something.

--
-- Jos Vos <***@xos.nl>
-- X/OS Experts in Open Systems BV | Phone: [phone]
-- Amsterdam, The Netherlands | Fax: [phone]

[marks]

Hi,

I had a similar challange. Our products are also not delivered as source.
We provide shell scripts for install and remove along with the package
content that includes pre-built binaries.

I felt I had to decide between two approaches in order to build rpm
packages of our products. The first choice was to modify all our product
source nodes (make files, adding "make install", etc.) to support building
of rpm packages from source code which the customer would never see. In
addition, every other product source node would require the same types of
edits/changes be performed again. Taking this course of action would take
extra time and create unnecessary life cycles for each product that needed
to support rpm. We only had one customer requesting rpm so I felt this
choice would result in more up front engineering effort than it was worth
and make it much harder to meet the desired schedule.

The second choice was to develop an rpm spec file for the needed products
that essentially used the existing product packages and their install
script from inside the rpm spec file. Most of our products are similar so
by developing a generic spec file that used variables at the top to set
product specifics (like product name, version, etc.) seemed the best way
to go.

What I ended up doing is using the %prep section of the spec file to setup
$RPM_BUILD_ROOT to contain the bare minimum linux filesystem content
(directories and files) required by our product install scripts. The
system directories and files that would get updated by our install script
needed to exist in advance of execution of the %install section described
below.

I then used the %install section to run our install script to install the
product in question to the $RPM_BUILD_ROOT. It helped that our install
scripts already supported relocatable root installations (i.e.
installations to a virtual root directory other than /). You should be
able to parameterize / in your install script to do the same thing.
Start by creating an empty directory, populate it with system directories
and files you know are needed by your install script, then try installing
your product to this directory. Once you know all the steps needed to
setup an empty directory as a virtual root directory for installation you
can adjust the %prep section to perform these steps automatically.

At the end of the %install section I removed any system files that were
updated by our install script. The sections of our install and remove
scripts that handled changes to system files were folded into the
appropriate %pre, %post, %preun and %postun sections of the rpm spec file.
You may also find that when rpmbuild runs it will complain about
unpackaged files located in $RPM_BUILD_ROOT as compared to the files in
your %files section. Some of the files may actually belong to your
package and need to be added to %files. Other files may not belong to
your package and are simply there to support installing your package to
$RPM_BUILD_ROOT in which case they would be candidates for removal from
$RPM_BUILD_ROOT at the bottom of the %install section after your install
script is done installing your product.

The %clean section simply cleans out $RPM_BUILD_ROOT. I took extra
precautions to make sure $RPM_BUILD_ROOT exists as a directory and is not
set to / before running "rm -rf $RPM_BUILD_ROOT". Although, if you build
your rpms as a jive user (not as root) as recommended at least your system
damage would be minimized if something went wrong. I decided to develop
as root since our install script required the user to be logged in as root
in order to execute. I used a freshly installed system to do my initial
rpm development so didn't care if something got whacked as it took less
than 40 minutes to wipe the disk and re-install (never needed to though).
Note: I initially did not use %clean at all as I needed to retain the
content of $RPM_BUILD_ROOT in order to correct problems while I developed
my process. I would manually wipe out $RPM_BUILD_ROOT when needed.

I did not use a %setup or %build section in my spec file.

What I ended up with was easily adapted to the other products that needed
to support rpm packages.

One of the items that surprised me about rpm that is worth mentioning
(shown via a script excerpt):

%preun
# rpm upgrades (-U) install the new package first then remove
# the old package which results in the old package's preun and
# postun scripts running last in an upgrade scenario. Therefore,
# check parameter 1 to see if this is an upgrade to a new version
# ($1 >= 2) versus a removal of the last installed version
# ($1 = 0). Skip the following if this is an upgrade so we
# don't break the new package installation.
if [ $1 -eq 0 ]
then
last package installed removal steps go here...
fi

The above is located in the %preun section of my spec file. Depending
on your requirements you may also need something similar in your %pre
section:

%pre
# $1 will be 1 if this is a first time rpm install and >= 2 if
# this is an upgrade (i.e. if there is already a package of the
# same name installed).
if [ $1 -le 1 ]
then
first time install pre-install steps go here...
fi


Welcome to the wonderful (and sometimes whacky) world of rpm. :-)


On Wed, 20 Jun 2007, Novick, Randy wrote:

> Date: Wed, 20 Jun 2007 15:38:34 -0500
> From: "Novick, Randy" <***@McKesson.com>
> Reply-To: RPM Package Manager <rpm-***@redhat.com>
> To: rpm-***@redhat.com
> Subject: ... binary RPM question ...
>
> Hi,
>
> This is my first post to the list, though I've been lurking for a few
> days.
>
> I have been reading Maximun RPM and a host of other howtos and I've got
> a .spec file in progress, but I'm confused about how to do something
> that appears to not fit so neatly into the documentation I've read.
>
> The challenge is this:
>
> We have our own build process that creates all the necessary object
> files on each of our UNIX platforms (there are three, and AIX and Linux
> are the two I'm interested in delivering via RPM). We cannot ship our
> sources to our customers, yet we would like to leverage the RPM database
> to track and manage what files get installed at the customer site.
>
> We have a packaging process that follows compilation which plucks all of
> the objects and library dependencies into a "runtime" directory that
> represents what the customer will have installed. This "runtime"
> directory is tarred up and gzipped and this is what we deliver to the
> customer as a release, also shipping a set of scripts alongside that
> handle the gunzipping and untarring of the deliverable, the creation of
> logdirs, the setup of config files and the tweaking of system settings
> necessary for getting things to run.
>
> What I want to do is to create an RPM that contains what I've compiled
> in the form of the gzipped tar I create, and also encapsulates the
> scripting necessary to untar, create dirs, setup configs and tweak
> settings in the .spec file.
>
> I'm stymied, though, as to how to use the rpmbuild dirs to achieve this.
> Which ones should I not use, if I'm not having the customer compile the
> code as a part of RPM installation? What steps in the .spec file can I
> skip? Do I get any advantage out of not pre-tarring-and-gzipping my
> build products? Which steps in the .spec file must I certainly use to
> contain my scripting?
>
> I've had a lot of experience with solaris packaging, so I understand
> preinstall, install, postinstall ... and preun, un and postun. Rpmbuild
> has got a lot more complexity, though, and appears to be more suited to
> open source deliveries than the kind of closed-source installs I'm
> trying to achieve.
>
> I'd appreciate some advice, and some help getting started.
> Thanks,
> -- R
>
>
>
>
>
> Randy Novick
>
> Software Configuration Management
>
> McKesson Provider Technologies
>
> ***@mckesson.com
>
> [phone]
>
>
>
> Confidentiality Notice: This email message, including any attachments,
> is for the sole use of the intended recipient(s) and may contain
> confidential and privileged information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all
> copies of the original message.
>
>
>

Thanks,

Mark Sincerbox
Adax, Inc.
1-510-548-7047 x129

[Jos Vos]

On Wed, Jun 20, 2007 at 05:54:35PM -0700, marks wrote:

> What I ended up doing is using the %prep section of the spec file to setup
> $RPM_BUILD_ROOT to contain the bare minimum linux filesystem content
> (directories and files) required by our product install scripts. The
> system directories and files that would get updated by our install script
> needed to exist in advance of execution of the %install section described
> below.
>
> I then used the %install section to run our install script to install the
> product in question to the $RPM_BUILD_ROOT. It helped that our install
> scripts already supported relocatable root installations (i.e.
> installations to a virtual root directory other than /). You should be
> able to parameterize / in your install script to do the same thing.
> Start by creating an empty directory, populate it with system directories
> and files you know are needed by your install script, then try installing
> your product to this directory. [...]

Better adapt the install script (if you can influence that) to do the
following:

- Create needed (system) directories if not already there (so that
you can always start with an empty buildroot).

- Have a option to *not* modify existing files (or do that always when
an alternative root is given, but this may be undesired when the
alternative root is a "real" root fs).

> [...]. I decided to develop
> as root since our install script required the user to be logged in as root
> in order to execute. [...]

Why not adapt the install script so that it can also handle non-root
installs *if* an alternative root is given?

--
-- Jos Vos <***@xos.nl>
-- X/OS Experts in Open Systems BV | Phone: [phone]
-- Amsterdam, The Netherlands | Fax: [phone]

[marks]

Hi,

inline...

On Thu, 21 Jun 2007, Jos Vos wrote:

> Date: Thu, 21 Jun 2007 09:52:22 +0200
> From: Jos Vos <***@xos.nl>
> Reply-To: RPM Package Manager <rpm-***@redhat.com>
> To: RPM Package Manager <rpm-***@redhat.com>
> Subject: Re: ... binary RPM question ...
>
> On Wed, Jun 20, 2007 at 05:54:35PM -0700, marks wrote:
>
> > What I ended up doing is using the %prep section of the spec file to setup
> > $RPM_BUILD_ROOT to contain the bare minimum linux filesystem content
> > (directories and files) required by our product install scripts. The
> > system directories and files that would get updated by our install script
> > needed to exist in advance of execution of the %install section described
> > below.
> >
> > I then used the %install section to run our install script to install the
> > product in question to the $RPM_BUILD_ROOT. It helped that our install
> > scripts already supported relocatable root installations (i.e.
> > installations to a virtual root directory other than /). You should be
> > able to parameterize / in your install script to do the same thing.
> > Start by creating an empty directory, populate it with system directories
> > and files you know are needed by your install script, then try installing
> > your product to this directory. [...]
>
> Better adapt the install script (if you can influence that) to do the
> following:
>
> - Create needed (system) directories if not already there (so that
> you can always start with an empty buildroot).
>
> - Have a option to *not* modify existing files (or do that always when
> an alternative root is given, but this may be undesired when the
> alternative root is a "real" root fs).
>
> > [...]. I decided to develop
> > as root since our install script required the user to be logged in as root
> > in order to execute. [...]
>
> Why not adapt the install script so that it can also handle non-root
> installs *if* an alternative root is given?

thanks for the comments...

In order to adapt/modify install scripts I would have to open every
product source node that needed to support rpm and start a development
cycle. If I were to do that I may as well modify all the make files to do
"make install". My goal was to not have to modify any existing product
files. I didn't want to start a new product development/test/release
cycle for the sole reason of adding rpm support. The course I have taken
will allow me to just include the new spec file in each product at a time
when that product is opened for development for other reasons.

>
> --
> -- Jos Vos <***@xos.nl>
> -- X/OS Experts in Open Systems BV | Phone: [phone]
> -- Amsterdam, The Netherlands | Fax: [phone]
>
> _______________________________________________
> Rpm-list mailing list
> Rpm-***@redhat.com
> https://www.redhat.com/mailman/listinfo/rpm-list
>

Thanks,

Mark Sincerbox

[Leland Ray]

>If you don't have a Source: tag, where do you get
your >files from?
>Out of the blue? From "what's there at that moment"?

If you are not using %prep (as itself, or as %setup),
the Source: tags only exist to tell rpmbuild what to
use when constructing a .src.rpm.

So the simple answer to the question is that there are
no source files in the case of constructing a binary
rpm; you already have the binaries, and the previous
build system gives you the layout.

Yes, I should have mentioned the problem of
inadvertantly including a system directory in %files,
which is a real danger if you use a wildcard.


Whether the original poster should use the tar/gzip
runtime directory as a source to rpm as you suggested,
or just use the final packaging stage of rpmbuild,
depends on his specific requirements.


____________________________________________________________________________________
Shape Yahoo! in your own image. Join our Network Research Panel today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7

[Jos Vos]

On Thu, Jun 21, 2007 at 07:28:38AM -0700, Leland Ray wrote:

> If you are not using %prep (as itself, or as %setup),
> the Source: tags only exist to tell rpmbuild what to
> use when constructing a .src.rpm.
>
> So the simple answer to the question is that there are
> no source files in the case of constructing a binary
> rpm; you already have the binaries, and the previous
> build system gives you the layout.

Building a binary RPM without a source RPM (in this case meaning an
RPM containing a tarball with binaries) is in general a bad idea,
as rebuilding the (binary) RPM is not posisble then.

Using files from outside the SOURCES directory is just "not done"
when building RPMs and that what's happening here.

--
-- Jos Vos <***@xos.nl>
-- X/OS Experts in Open Systems BV | Phone: [phone]
-- Amsterdam, The Netherlands | Fax: [phone]

[Hiren Patel]

i completely agree so far to Jos's recommendations.
do it the proper way from the outset, you will not regret it later.
best practices don't exist for nothing.

On Thu, 2007-06-21 at 16:37 +0200, Jos Vos wrote:
> On Thu, Jun 21, 2007 at 07:28:38AM -0700, Leland Ray wrote:
>
> > If you are not using %prep (as itself, or as %setup),
> > the Source: tags only exist to tell rpmbuild what to
> > use when constructing a .src.rpm.
> >
> > So the simple answer to the question is that there are
> > no source files in the case of constructing a binary
> > rpm; you already have the binaries, and the previous
> > build system gives you the layout.
>
> Building a binary RPM without a source RPM (in this case meaning an
> RPM containing a tarball with binaries) is in general a bad idea,
> as rebuilding the (binary) RPM is not posisble then.
>
> Using files from outside the SOURCES directory is just "not done"
> when building RPMs and that what's happening here.
>
--

Hiren Patel | Ops Specialist | ISS Infrastructure | Telkom
E-Mail: ***@telkom.co.za Office: [phone] | Fax: [phone]
3299 | Cell: [phone]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This e-mail and its contents are subject to the Telkom SA Limited
e-mail legal notice available at
http://www.telkom.co.za/TelkomEMailLegalNotice.PDF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Novick, Randy]

Thank you Hiren and Jos for your opinions.

I understand and appreciate where they come from. I am, however, legally
bound to not ship our source code, in any form, at any time, to any
customer, by any vehicle. The cost to the company, were I to do so,
would be in the billions of dollars, and the cost to my own career would
be incalculable, to say nothing of my own sense of integrity.

Thanks again,
-- R

-----Original Message-----
From: Hiren Patel [mailto:***@telkom.co.za]
Sent: Thursday, June 21, 2007 8:55 AM
To: RPM Package Manager
Subject: Re: ... binary RPM question ...


i completely agree so far to Jos's recommendations.
do it the proper way from the outset, you will not regret it later.
best practices don't exist for nothing.

On Thu, 2007-06-21 at 16:37 +0200, Jos Vos wrote:
> On Thu, Jun 21, 2007 at 07:28:38AM -0700, Leland Ray wrote:
>
> > If you are not using %prep (as itself, or as %setup), the Source:
> > tags only exist to tell rpmbuild what to use when constructing a
> > .src.rpm.
> >
> > So the simple answer to the question is that there are no source
> > files in the case of constructing a binary rpm; you already have the

> > binaries, and the previous build system gives you the layout.
>
> Building a binary RPM without a source RPM (in this case meaning an
> RPM containing a tarball with binaries) is in general a bad idea, as
> rebuilding the (binary) RPM is not posisble then.
>
> Using files from outside the SOURCES directory is just "not done"
> when building RPMs and that what's happening here.
>
--

Hiren Patel | Ops Specialist | ISS Infrastructure | Telkom
E-Mail: ***@telkom.co.za Office: [phone] | Fax: [phone]
3299 | Cell: [phone]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This e-mail and its contents are subject to the Telkom SA Limited e-mail
legal notice available at
http://www.telkom.co.za/TelkomEMailLegalNotice.PDF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Jos Vos]

On Thu, Jun 21, 2007 at 10:10:54AM -0500, Novick, Randy wrote:

> I understand and appreciate where they come from. I am, however, legally
> bound to not ship our source code, in any form, at any time, to any
> customer, by any vehicle. The cost to the company, were I to do so,
> would be in the billions of dollars, and the cost to my own career would
> be incalculable, to say nothing of my own sense of integrity.

I have never suggested to let the source RPM contain actual source
code, but the input files (in your case tarballs with binaries)
are tagged "Source". The recommendation is to always create source
RPMs, even when the "sources" are in fact the binaries.

--
-- Jos Vos <***@xos.nl>
-- X/OS Experts in Open Systems BV | Phone: [phone]
-- Amsterdam, The Netherlands | Fax: [phone]

[Hiren Patel]

the source and binary rpms are separate, you could still use Jos's
suggestions, create a source and binary rpm, distribute the binary to
customers and keep the source rpm undisclosed.

On Thu, 2007-06-21 at 10:10 -0500, Novick, Randy wrote:
> Thank you Hiren and Jos for your opinions.
>
> I understand and appreciate where they come from. I am, however, legally
> bound to not ship our source code, in any form, at any time, to any
> customer, by any vehicle. The cost to the company, were I to do so,
> would be in the billions of dollars, and the cost to my own career would
> be incalculable, to say nothing of my own sense of integrity.
>
> Thanks again,
> -- R
>
> -----Original Message-----
> From: Hiren Patel [mailto:***@telkom.co.za]
> Sent: Thursday, June 21, 2007 8:55 AM
> To: RPM Package Manager
> Subject: Re: ... binary RPM question ...
>
>
> i completely agree so far to Jos's recommendations.
> do it the proper way from the outset, you will not regret it later.
> best practices don't exist for nothing.
>
> On Thu, 2007-06-21 at 16:37 +0200, Jos Vos wrote:
> > On Thu, Jun 21, 2007 at 07:28:38AM -0700, Leland Ray wrote:
> >
> > > If you are not using %prep (as itself, or as %setup), the Source:
> > > tags only exist to tell rpmbuild what to use when constructing a
> > > .src.rpm.
> > >
> > > So the simple answer to the question is that there are no source
> > > files in the case of constructing a binary rpm; you already have the
>
> > > binaries, and the previous build system gives you the layout.
> >
> > Building a binary RPM without a source RPM (in this case meaning an
> > RPM containing a tarball with binaries) is in general a bad idea, as
> > rebuilding the (binary) RPM is not posisble then.
> >
> > Using files from outside the SOURCES directory is just "not done"
> > when building RPMs and that what's happening here.
> >
--

Hiren Patel | Ops Specialist | ISS Infrastructure | Telkom
E-Mail: ***@telkom.co.za Office: [phone] | Fax: [phone]
3299 | Cell: [phone]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This e-mail and its contents are subject to the Telkom SA Limited
e-mail legal notice available at
http://www.telkom.co.za/TelkomEMailLegalNotice.PDF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[Leland Ray]

In product builds that must span multiple
architectures and multiple packaging methods it is
quite common that rpmbuild not control the build
process, and that the build system does not create a
usable .src.rpm.

The question of whether to use rpm's source
abstraction or not is one of requirements, and not
technical. Binding the rpm creation to the build as
per my original suggestion allows for tight
integration with the upstream build. In that case, the
only valid trigger to start packaging is the
conclusion of the upstream build. The requirement is
that any failure of packaging causes the build to
abort. The pattern also improves reliability in terms
of version tracking.

The proposal of using the built binaries as sources
allows for packaging to be done separately. Sometimes
the procedure is to build and perform preliminary
testing before packaging, or sometimes the product is
released on different platforms at different times, so
the rpm packaging is not always built and tested.

Both abstractions are useful for different products,
and I've certainly used them both.



____________________________________________________________________________________Ready for the edge of your seat?
Check out tonight's top picks on Yahoo! TV.
http://tv.yahoo.com/