|
|
When building your JE application, you should consider how you will run recovery. If you are building a single threaded, single process application, it is fairly simple to run recovery when your application first opens its environment. In this case, you need only decide if you want to run recovery every time you open your application (recommended) or only some of the time, presumably triggered by a start up option controlled by your application's user.
However, for multi-threaded and multi-process applications, you need to carefully consider how you will design your application's startup code so as to run recovery only when it makes sense to do so.
If your application uses only one environment handle, then handling recovery for a multi-threaded application is no more difficult than for a single threaded application. You simply open the environment in the application's main thread, and then pass that handle to each of the threads that will be performing JE operations. We illustrate this with our final example in this book (see Transaction Example for more information).
Alternatively, you can have each worker thread open its own environment handle. However, in this case, designing for recovery is a bit more complicated.
Generally, when a thread performing database operations fails or hangs, it is frequently best to simply restart the application and run recovery upon application startup as normal. However, not all applications can afford to restart because a single thread has misbehaved.
If you are attempting to continue operations in the face of a misbehaving thread, then at a minimum recovery must be run if a thread performing database operations fails or hangs.
Remember that recovery clears the environment of all outstanding locks, including any that might be outstanding from an aborted thread. If these locks are not cleared, other threads performing database operations can back up behind the locks obtained but never cleared by the failed thread. The result will be an application that hangs indefinitely.
To run recovery under these circumstances:
Suspend or shutdown all other threads performing database operations.
Discarding any open environment handles. Note that attempting to gracefully close these handles may be asking for trouble; the close can fail if the environment is already in need of recovery. For this reason, it is best and easiest to simply discard the handle.
Open new handles, running recovery as you open them. See Normal Recovery for more information.
Restart all your database threads.
A traditional way to handle this activity is to spawn a watcher thread that is responsible for making sure all is well with your threads, and performing the above actions if not.
However, in the case where each worker thread opens and maintains its own environment handle, recovery is complicated for two reasons:
For some applications and workloads, it might be worthwhile to give your database threads the ability to gracefully finalize any on-going transactions. If this is the case, your code must be capable of signaling each thread to halt JE activities and close its environment. If you simply run recovery against the environment, your database threads will detect this and fail in the midst of performing their database operations.
Your code must be capable of ensuring only one thread runs recovery before allowing all other threads to open their respective environment handles. Recovery should be single threaded because when recovery is run against an environment, it is deleted and then recreated. This will cause all other processes and threads to "fail" when they attempt operations against the newly recovered environment. If all threads run recovery when they start up, then it is likely that some threads will fail because the environment that they are using has been recovered. This will cause the thread to have to re-execute its own recovery path. At best, this is inefficient and at worst it could cause your application to fall into an endless recovery pattern.
Frequently, JE applications use multiple processes to interact with the databases. For example, you may have a long-running process, such as some kind of server, and then a series of administrative tools that you use to inspect and administer the underlying databases. Or, in some web-based architectures, different services are run as independent processes that are managed by the server.
In any case, recovery for a multi-process environment is complicated for two reasons:
In the event that recovery must be run, you might want to notify processes interacting with the environment that recovery is about to occur and give them a chance to gracefully terminate. Whether it is worthwhile for you to do this is entirely dependent upon the nature of your application. Some long-running applications with multiple processes performing meaningful work might want to do this. Other applications with processes performing database operations that are likely to be harmed by error conditions in other processes will likely find it to be not worth the effort. For this latter group, the chances of performing a graceful shutdown may be low anyway.
Unlike single process scenarios, it can quickly become wasteful for every process interacting with the databases to run recovery when it starts up. This is partly because recovery does take some amount of time to run, but mostly you want to avoid a situation where your server must reopen all its environment handles just because you fire up a command line database administrative utility that always runs recovery.
JE offers you two methods by which you can manage recovery for multi-process JE applications. Each has different strengths and weaknesses, and they are described in the next sections.
Before continuing, it is worth noting that the following sections describe recovery processes than can result in one process running recovery while other processes are currently actively performing database operations.
When this happens, the current database operation will abnormally fail, indicating a DB_RUNRECOVERY condition. This means that your application should immediately abandon any database operations that it may have on-going, discard any environment handles it has opened, and obtain and open new handles.
The net effect of this is that any writes performed by unresolved transactions will be lost. For persistent applications (servers, for example), the services it provides will also be unavailable for the amount of time that it takes to complete a recovery and for all participating processes to reopen their environment handles.
One way to handle multi-process recovery is for every process to "register" its environment. In doing so, the process gains the ability to see if any other applications are using the environment and, if so, whether they have suffered an abnormal termination. If an abnormal termination is detected, the process runs recovery; otherwise, it does not.
Note that using process registration also ensures that recovery is serialized across applications. That is, only one process at a time has a chance to run recovery. Generally this means that the first process to start up will run recovery, and all other processes will silently not run recovery because it is not needed.
To cause your application to register its environment, you specify the DB_REGISTER flag when you open your environment. Note that you must also specify DB_RECOVER or DB_RECOVER_FATAL for your environment open. If during the open, JE determines that recovery must be run, this indicates the type of recovery that is run. If you do not specify either type of recovery, then no recovery is run if the registration process identifies a need for it. In this case, the environment open simply fails by returning DB_RUNRECOVERY.
Be aware that there are some limitations/requirements if you want your various processes to coordinate recovery using this registration process:
There can be only one environment handle per environment per process. In the case of multi-threaded processes, the environment handle must be shared across threads.
All processes sharing the environment must use registration. If registration is not uniformly used across all participating processes, then you can see inconsistent results in terms of your application's ability to recognize that recovery must be run.
You can not use this mechanism with the failchk() mechanism described in the next section.
For very large and robust multi-process applications, the most common way to ensure all the processes are working as intended is to make use of a watchdog process. To assist a watchdog process, Sleepycat offers a failure checking mechanism.
When a thread of control fails with open environment handles, the result is that there may be resources left locked or corrupted. Other threads of control may encountered these unavailable resources quickly or not at all, depending on data access patterns.
In any case, the Sleepycat failure checking mechanism allows a watchdog to detect if an environment is unusable as a result of a thread of control failure. It should be called periodically (for example, once a minute) from the watchdog process. If the environment is deemed unusable, then the watchdog process is notified that recovery should be run. It is then up to the watchdog to actually run recovery. It is also the watchdog's responsibility to decide what to do about currently running processes before running recovery. The watchdog could, for example, attempt to gracefully shutdown or kill all relevant processes before running recovery.
Note that failure checking need not be run from a separate process, although conceptually that is how the mechanism is meant to be used. This same mechanism could be used in a multi-threaded application that wants to have a watchdog thread.
To use failure checking you must:
Provide an is_alive() call back using the DB_ENV->set_isalive() method. JE uses this method to determine whether a specified process and thread is alive when the failure checking is performed.
Possibly provide a thread_id callback that uniquely identifies a process and thread of control. This callback is only necessary if the standard process and thread identification functions for your platform are not sufficient to for use by failure checking. This is rarely necessary and is usually because the thread and/or process ids used by your system cannot fit into an unsigned integer.
You provide this callback using the DB_ENV->set_thread_id() method. See the API reference for this method for more information on when setting a thread id callback might be necessary.
Call the DB_ENV->failchk() method periodically. You can do this either periodically (once per minute, for example), or whenever a thread of control exits for your application.
If this method determines that a thread of control exited holding read locks, those locks are automatically released. If the thread of control exited with an unresolved transaction, that transaction is aborted. If any other problems exist beyond these such that the environment must be recovered, the method will return DB_RUNRECOVERY.
Note that this mechanism should not be mixed with the process registration method of multi-process recovery described in the previous section.