On 11/28/06, Tony Nelson <***@georgeanelson.com> wrote:
> I haven't seen this come through.
>
> At 8:48 AM -0500 11/24/06, Jeff Johnson wrote:
> >On 11/22/06, Tony Nelson <***@georgeanelson.com> wrote:
> >> The RPM database should be verified more often than it is now. I've posted
> >> on this topic to the fedora-devel list.
> >>
> >
> >Perhaps.
> >
> >Using --verifydb (or equivalently, running /usr/lib/rpm/rpmdb_verify, the
> >preferred solution these days) does not do data checks, only structural
> >checks on the database.
>
> True. It would be a good thing if RPM did such checks.
>
We disagree, at least on the frequency of checking and the amount of automation.
But there is nothing stopping you from using --verifydb, or rpmdb_verify, or
configuring rpm to verify every index on every close if you desire.
Choosing the default behavior configured, as always in rpm, is where most
of the disagreement lies.
>
> >(aside) rpm-4.0.2 used to do --verifydb on every close. I don't
> >recall any problem that was meaningfully detected and solved
> >by verifying more often. FWIW, the macros to verify a dabase
> >on every close are likely still present and functional in current
> >rpm.
>
> "More often" --> "that often"?
>
Yes, that is the sense of my reply.
>
> >Note that running --verifydb can/will create locks, which is really
> >the source of recent reported problems, and running --verifydb
> >is likelier to create more, rather than fewer, problem reports imho.
>
So create problem reports. The current problem reports have to do
with stale locks and __db cache corruption, not with Packages or --verifydb.
> I hope rpm-verifydb will create more problem reports. Currently, problems
> in the RPM database can go unnoticed for a long time, until disaster
> strikes.
>
If a tree falls in the forest, who hears the noise?
> I chose "rpm --verifydb" over some Berkely DB tool as I expect that rpm
> already does proper locking of its database during checks (modulo any
> kernel bugs).
>
RPM with concurrent access uses exactly Berkely DB locks, there really
is no meaningful difference between --verifydb and rpmdb_verify.
The external executable is preferred to simplify rpm options (there are
far too many) and to provide better doco (of which there is far too little)
for verifying an rpmdb.
>
> >The data contained in Packages is signature/digest checked. A digest check
> >is about the strongest data integrity check possible.
> >
> >The only other data in an rpmdb is join keys and index keys, and
> >that data is regenerated whenever --rebuilddb is run.
>
> I take it that all the data used by "--rebuilddb" is in the Packages file,
> and that the Packages file is carefully checked by "--rebuilddb"?
>
I have no idea what you mean by "carefully". If you ask more specifically,
I might be able to say what is and isn't checked.
>
> >> With FC6 there has been a spate of RPM database corruption. It happened to
> >> me: though there may have been incipient corruption in my FC5, after
> >> --rebuilddb and upgrading successfully I found more corruption later. This
> >> brings up that the RPM database is just assumed to work, but isn't being
> >> checked until it falls over.
> >>
> >
> >No there has not been a "spate of RPM database corruption".
>
> How do you know? 8-) After all, who's checking now?
>
Ultimately, noone knows.
Meanwhile, I have years of experience diagnosing and fixing rpmdb
problems. I am not seeing or hearing that headers were damaged in
databases (which is my definition of "corruption"). YMMV as always.
>
> >Yes there have been a number of reports of stale locks and cache
> >(not database) incoherency. And there are a few other segfaults
> >being reported against rpm.
> >
> >But feel free to describe rpm problems however you wish.
>
> OK.
>
>
> >> I propose that the RPM database should be verified on a regular basis. I
> >> have written a utility, rpm_verify_db, to automatically verify and repair
> >> the RPM database, via a daily cron job. Reports of errors are syslog'd,
> >> emailed to root, and shown by logwatch. It could be incorporated into the
> >> RPM package, or even Yum. It can be found at
> >> <http://georgeanelson.com/rpm-verifydb.htm>.
> >>
> >
> >rpm used to do --verifydb, stopped because no problems were prevented or
> >solved by verifying.
> >
> >> I propose that Anaconda should check the RPM database before starting an
> >> upgrade to an existing installation. Checking takes under a minute on my
> >> system, so it should not be objectionable. Anaconda should offer to repair
> >> a damaged RPM database (if the Package file is OK) before proceeding with
> >> the installation.
> >>
> >
> >Anaconda used to do a --rebuilddb, stopped because no problems were
> >being usefully solved.
>
> Hmm, that would have solved the problem I reported in RH BZ 215127, and
> probably the OP's problem as well. I don't know how to find out if it
> would not have helped anyone else.
>
Then ask anaconda (or yum) to do run rpmdb_verify or --verifydb or
call the python method ts.verifyDB() as needed. When an rpmdb is
verified is not solvable in rpmlib
(except by verifying on every close, been there, done that, dinna help).
>
> >> I suggest that the --verifydb command should not be undocumented in RPM and
> >> its manpage. This seems to be on purpose, but I think it is a mistake.
> >>
> >
> >The option is undocumented because using /usr/lib/rpm/rpmdb_verify (which
> >is exactly the same operation) is the preferred means of database repair.
>
> Where is rpmdb_verify documented? "man rpmdb_verify" doesn't find
> anything. "apropos rpm" | grep 'verify'" doesn't find anything either. It
> doesn't seem to be in "man rpm".
>
Look for "db_verify" in the Berkely DB utilities doco, which is also included
in the db4 package last I checked.
>
> >rpmdb_verify is exactly the Sleepycat distributed utility linked against rpm
> >rather than system libraries, and is quite well documented at sleepycat.com.
>
> RPM users would benefit from having that documented some place RPM (other
> than this list). "man rpm" is where they would expect to find it.
>
"expect" is in the eye of the beholder.
>
> >> I would like some feedback about these proposals. If they are acceptable I
> >> will file RFE bugs on them.
> >>
> >> My knowlege of things RPM is superficial. It would be a good idea to have
> >> my proposed verification and repair methods criticised by authentic RPM
> >> developers.
> >>
> >
> >None of the above should be considered criticism, just history.
> >
> >There are two needs if one wants to protect against rpmdb data loss.
> >
> >The most important is saving a copy of /var/lib/rpm/Packages routinely.
> >All other information in an rpmdb can be regenerated from a reasonably
> >recent copy of Packages. And in most cases a depsolver like
> >yum/smart/apt/poldek
> >will reinstall the packages that have changed since the last copy of Packages
> >was saved.
>
> OK, though Packages is too large to save every day, and just keeping the
> last one would mean that the sysadmin would need to notice the problem
> before that last good copy of Packages was overwritten. (I have an RPM
> database instance that has corruption but functioned normally most of the
> time, though not during an Anacoda upgrade to FC6. See RH BZ 215127.) It
> would seem that it would be better to only save good copies of the Packages
> file, and to report to a responsible sysadmin that there is an issue (if
> only there were such a sysdamin for most systems). I don't know how to do
> that with "rpm --rebuilddb", but I do know how to do that with "rpm
> --verifydb".
>
Too large is a different problem, try "man rdiff" if you want an easy
incremental
backup.
>
> >The other important need is to have a find-like utility to reconstruct an
> >rpmdb
> >using only md5 digests of installed files for those people who are not saving
> >a copy of Packages ;-)
>
> /No one/ saves a copy of Packages, as such an implementation detail is
> RPM's responsibility, and it does not do it. Don't blame the users for
> omissions!
We disagree. If you want to protect your data, you need to take precautions.
RPM cannot do reliable backups for all possible users under all cases.
Meanwhile, there is nothing stopping you or anyone else from doing whatever
you wish to achieve reliability.
73 de Jeff