Ipqbdb malfunctions

Malfunctions may arise because of how the Concurrent Data Store (CDB) 
model works. When a program opens the database environment, some 
__db.00? files are created in that directory. Each program has an 
option to cleanup or not this stuff. By default, ibd-parse and 
ibd-judge attempt to cleanup, the other programs don't. Cleaning up is 
not strictly necessary. However, failed attempts to cleanup may be 
logged as

  ibd-XXXX: not cleaning up /var/lib/ipqbdb: Device or resource busy

That is fine if there was another program running on that. If ibd-XXXX 
was the last program exiting, that line indicates a crash had happened.

In case a program crashes it will not close tables nor any write-lock 
it might have acquired. This causes harm: Access to the database will 
be denied and programs frozen. Use the db_deadlock utility (e.g. 
db_deadlock from the db-util debian pkg) to diagnose such a situation. 
Access errors may get logged after db_deadlock unblocks frozen 
programs.

Also useful for diagnosing:
  fuser -v /var/lib/ipqbdb/*
  lsof +D /var/lib/ipqbdb
  lslocks | grep descr_db.lock
  gdb -q -p <stuck-or-daemon-pid> -ex 'set pagination off' -ex 'thread apply all bt' -ex detach -ex quit

Since version 2.03, ipqbdb uses flock on file <DB home>/descr_db.lock 
around opening the descriptions file itself and around getting a write
cursor to update the catch count after a successful block.  In fact,
it was found that a storm of ibd-ban's while ibd-parse is running
cause a hang at an internal CDB call generated while opening the
descriptions database or acquiring a write cursor for it.  It seems
to be a Berkeley DB quirk.

If the recovery won't work, to restore, you should

  STOP AND RESTART:
  1. killall -e ibd-parse ibd-judge, or equivalent action,
  2. rm /var/lib/ipqbdb/__db.00?, or equivalent action,
  3. rm /var/lib/ipqbdb/clients/*
  4. optionally run dbX.Y_verify /var/lib/ipqbdb/*.db, or similar,
  5. restart ibd-parse and ibd-judge as needed, and
  6. restart the logger or whoever was writing to ibd-parse's socket.

Keep in mind that your installation can use a directory different than
/var/lib/ipqbdb; run ibd-config to check.  The "Upgrading" chapter in
INSTALL has a much more weathered list for restarting.

Keep in mind that while ibd-judge is not running your iptables rules 
will act as if they always get a DROP. To disable filtering remove the 
rules.

Avoid to delete database files while programs are using them. All ibd- 
programs open the database environment as part of their initialization 
and close it right before exiting. Keep in mind that all the .db files 
that a program works with must live in the same directory.

--

Once in a blue moon, receiving queued packets fails.  With the older 
version of the libnetfilter-queue library, ibd-judge froze strangely, 
after attempting to close and reopen the queue(s).  It is recommended 
to use libnetfilter-queue version 0.17.  It may still happen to find a 
log line of

  ibd-judge: break loop after recv returns -1 (No buffer space available)

The daemon cleans up and restarts the loop in this case.  The actual 
cause of the error is still being investigated, but with the newer 
libnetfilter-queue library the daemon continues to work.

