CH A P T E R

Unit Server High Availability and AVM Protection
The unit high availability and AVM protections architecture ensures continuous availability of Cisco ANA functionality by detecting and recovering from a wide range of hardware and software failures. The distributed design of the system enables the impact radius caused by a single fault to be confined. This prevents all types of faults from setting into motion the domino effect, which can lead to a crash of all the management services. These topics describes how you can use Cisco ANA for unit redundancy and process protection:
Overview of Unit High Availability, page 17-1 Configuring Cisco ANA Units for High Availability, page 17-8 Managing the Watchdog Protocol, page 17-12 High Availability Registry Settings, page 17-13
For information on high availability for gateway servers, see Using Gateway High Availability, page 18-1.
Overview of Unit High Availability
High availability of the server backbone is achieved at several complementary levels. For example:
NEBS-3 compliant carrier-class server hardware. Watchdog within each unit, responsible for monitoring and, if necessary, automatically reloading failed processes. N+m warm standby protection for unit groups. Watchdog Protocol for AVM Protection, page 17-1 Unit N+m High Availability, page 17-2
See the following topics for more information:
Watchdog Protocol for AVM Protection
The watchdog protocol monitors the AVM processes to make sure any AVMs that have failed are restarted. This is called AVM protection and the GUI, the watchdog protocol is controlled by the AVM Protection check box. Each unit executes several processes: one control process and several AVM processes that execute VNEs. Each process within the unit is completely independent. The isolation
Cisco Active Network Abstraction 3.7.2 Administrator Guide OL-23134-01
Chapter 17 Overview of Unit High Availability
concept is tailored throughout the design so that a failure of a single process does not affect other processes on the same machine. The exact number of processes on each unit depends on the capacity and computational power of the unit. The control process executes a watchdog protocol, which continuously monitors all other processes on the unit. This watchdog protocol requires each AVM process to continuously handshake with the control process. A process that fails to handshake with the control process after a number of times is automatically cancelled and reloaded. The dynamic design of the control process implements runtime adaptation and escalation. The escalation procedure moves the AVM to suspended mode; that is, the process is suspended. An example of an escalation procedure is to stop reloading a process that has crashed more than n times within a given period, because it is suspected of having a recurring software problem. The reload process is local to the unit, and thus very rapid, with a minimal amount of downtime. In many cases the process can use its previous cache information (temporary persistency used to improve performance), once the stuck process id detected, reloading the process takes only a few seconds with no data loss. This is the case for user-created AVMs that are hosting VNEs. However, for reserved AVMs that perform special function in Cisco ANA, some data loss will occur. All watchdog activity is logged and an alarm is generated and sent when the watchdog reloads a process.
An alarm persistency mechanism enables the system to clear alarms that relate to events that occurred while a VNE, an AVM, a unit, or the whole system was down, thus preserving system integrity. For more information about alarm persistency, see Chapter 26, VNE Persistency Mechanism. All watchdog protocol parameters, such as pulse interval and retry times, are configurable in the registry. The higher these parameter values are, the longer the AVM or unit failure lasts, but this increases the certainty that a failure has actually occurred. Configuring these parameters with lower values may shorten the AVM or unit recovery, but might result in a false positive which could unnecessarily restart an AVM or revert to a standby unit when the AVM is just busy or the unit is processing a heavy load of data. For information on these registry settings, see High Availability Registry Settings, page 17-13.
Unit N+m High Availability
The clustered N+m unit high availability mechanism uses the Cisco ANA fabric is designed to handle the failure of a unit. Such failures include hardware failures, operating system failures, power failures, and network failures, which disconnect a unit from the Cisco ANA fabric. Unit availability is established in the gateway, running a protection manager process, which continuously monitors all the units in the network. Once the protection manager detects a unit that is malfunctioning, it automatically signals one of the standby servers in its cluster to load the configuration of the faulty unit (from the system registry), taking over all of its managed network elements. This design provides many possibilities for trading off protection and resources. These possibilities range from segmenting the network into clusters without any extra machines, to having a warm-swappable empty unit for each unit in the setup. When a unit is configured, it can be designated as being an active or standby unit. Using the GUI, you can designate a group of active units and a standby unit to be the members of a protection group, giving the group the name of your choice. A protection group can have multiple standby units, and you can define more than a single protection group. A unit switchover results in the unavoidable loss of information. The impact depends on how long the unit is down, and the functions the unit performed.

Cisco Active Network Abstraction 3.7.2 Administrator Guide

OL-23134-01

Chapter 17
Unit Server High Availability and AVM Protection Overview of Unit High Availability
Figure 17-1 shows a protection group (cluster) of units controlled by a gateway with one unit configured as the standby for the protection group.
Figure 17-1 Cisco ANA Protection GroupsExample

Cisco ANA Gateway

Cisco ANA Unit
Cisco ANA Cisco ANA Unit Unit (standby) (standby)

Cisco ANA Unit edge-pg

Cisco ANA Unit (standby)

default-pg

In the example configuration, when the gateway determines that one of the units in the protection group has failed, it notifies the standby unit of the protection group to immediately load the configuration of the failed unit. The standby unit loads the configuration of the failed unit, including all AVMs and VNEs, and functions as the failed unit. However, after a switchover in the protection group illustrated in Figure 17-1, the protection group would no longer have a standby unit. Therefore we recommend that you have two standby units per cluster. In this case, if a unit fails, another standby unit is still available. Because events are recorded in the Cisco ANA EventVision, you can check for the specific problem and take action to bring the failed unit up again. When the failed unit becomes operational, you can decide whether to configure it as the new standby unit or to reinstate it to the protection group and configure another unit as the standby unit.
AVM 100 and Unit High Availability
You can configure AVM 100 to run on a unit instead of the gateway. If the unit is also configured with high availability, the AVM 100 on the standby unit will drop all events because it is not running. This is by design; it should not start until a switchover occurs. The standby unit contains a port watchdog script that listens for events on the units Syslog and SNMP ports. The script prevents unnecessary ICMP unreachable messages being sent back to the network. If a switchover occurs, the standby unit and AVM 100 will start, and the watchdog script releases the ports. When the original unit comes back up, the standby AVM 100 goes back down, and the watchdog script recommences listening on the standby units Syslog and SNMP ports.
Recommendations for Configuring High Availability
Keep the following guidelines in mind when configuring protection groups:
Protection groups should be designed according to geography. For heavily loaded protection groups, add an additional standby unit. Units (active and standby) should not be assigned to more than one protection group.
Estimating the Impact of Unit or AVM Failures
When a failure occurs in a unit or AVM, the length of time that the system is down depends on the type of failure, how long it takes to detect that the component is not working, and the length of the recovery period (during which the unit or AVM reloads and the system begins to function normally again). Three types of failure can occur, as described in these topics:

Impact of Catastrophic AVM Process Failure, page 17-4 Impact of AVM Timeouts and Restarts, page 17-6 Impact of Unit Timeouts and Switchovers, page 17-7
Impact of Catastrophic AVM Process Failure
Each AVM has a log file which is constantly monitored by a Perl process for log messages about catastrophic failures, such as AVM processes running out of memory. When such a failure occurs, the Perl process restarts the AVM almost immediately, so the mean time to repair (MTTR) is based on the AVM loading life cycle. Table 17-1 describes the impact on different AVMs when experiencing such a failure.
Table 17-1 Catastrophic Process Failure Impact on AVMs
AVM Process AVM 0 (Switch/High Availability) AVM 11 (Gateway)
Results of AVM Failure Loss of messages to and from the machine.
Average Time To Repair Failed AVM 1 minute to reach bootstrap.
Degree of Impact to System if AVM Fails High. Messages are constantly being sent and received in the system. High. AVM 11 handles Oracle communication and various gateway functions such as alarm processing.
Loss of persistence information 6-10 minutes to reach bootstrap. for faults (except for the I persistency information handled by AVM 25 and AVM 100). No user authentication will be performed on gateway connections, and GUI clients will lose gateway connectivity.

Table 17-1

Catastrophic Process Failure Impact on AVMs (continued)
AVM Process AVM 25 (Event Persistence)

Results of AVM Failure

Average Time To Repair Failed AVM
Degree of Impact to System if AVM Fails High. Network events are constantly processed in a live, scaled system.
Loss of persistence information 1 minute to reach and new tickets for actionable bootstrap. network events that are processed while AVM 25 is down. When it comes up, new events that correlate to lost events will be persisted but will not be associated with a ticket until the integrity process identifies the broken chains (due to lost events) and opens new tickets. Results:

AVM 66 (Workflow)

Running workflows would abort. Scheduled workflows would not run. Templates would not be deployed.
1 minute to reach bootstrap, but with large number of workflows in the system, this may increase.
Low, because AVM 66 should sustain a large number of executed workflows (per the system limitations). Templates would need to be redeployed and aborted workflows would need to be rerun. Check the Provisioning events in Cisco ANA EventVision to verify what ran prior to failure, and then issue an rollback (no automatic rollback is done).

AVM 77 (Configuration and Image Management)
Loss of device configuration changes. Configuration changes will not be backed up to the archive during down time.
10 minutes for DM High, because configuration change notifications can happen all the time. server startup and bundle deployment, plus time to fetch all configurations for managed devices. Low, because registry modifications are made only when the VNE is first loaded into the system. Modifications are rarely made while the system is up and running. For the first 30 minutes after AVM 99 has started, there is no system monitoring for unit high availability. This allows the system enough time to get up and running

AVM 99 (Management)

Loss of registry notifications on 1 minute to reach changes made to golden source bootstrap. registry.
AVM 100 (ANA Event Collector)
Loss of traps and syslogs from devices, including raw event persistency.
1 minute to reach bootstrap, plus time for all the VNEs to register again for traps and syslogs. Normally a matter of minutes.
High, because raw events from devices are constantly received in a live, scaled system. Only devices registered to the failed AVM 100 are affected. No events will be handled during downtime. See AVM 100 and Unit High Availability, page 17-3. (Raw event persistency is recovered before events are forwarded to the VNEs.)
AVM Process AVM 101-999 (User-defined AVMs)
Results of AVM Failure Loss of management to a section of devices managed by the AVM; alarm state inconsistencies (user will have to clear tickets).
Degree of Impact to System if AVM Fails
High (but only for a period of one minute), because 1 minute to reach bootstrap, plus time no raw events sent to the VNEs can be processed when the AVM is down. to load the VNEs (depending on number, type, services, etc.).
Impact of AVM Timeouts and Restarts
Each AVM is constantly monitored by the management AVM (AVM 99) using a watchdog protocol pulse message sent to the AVM at preconfigured intervals. When the AVM fails to respond to the pulse message after a preconfigured number of attempts, the management AVM restarts the process. The management process also keeps a history of the number of times it has restarted the AVM. When it reaches the maximum number of preconfigured restart times, the management AVM stops restarting the AVM because this indicates a serious problem with the AVM. Each restart is logged as a System event (except when AVM 11 is restarted, because this AVM handles all persistency). Failures on AVMs in the system are measured in a way similar to that used for catastrophic process failures (see Table 17-1), with the addition of the watchdog protocol overhead. This is measured by the pulse interval multiplied by the number of restart attempts.

The maximum number of preconfigured restart times is five, after which the management process does not try to reload the AVM. It takes approximately one minute for the system to detect that an AVM (including AVM 100) is not working. The recovery period during which an AVM (including AVM 100) reloads and the system starts to function normally again is approximately five minutes, depending on the number of VNEs per AVM and the complexity of each.
Figure 17-2 provides a typical example of how unit high availability timer parameters work while monitoring AVMs.
If you are using gateway high availability, note that there is no overlapping between the processes that AVM 99 monitors that are illustrated in Figure 17-2, and the process that Veritas Cluster Manager monitors (the ANAGateway Veritas agent). For an illustration, see Figure 18-7 on page 18-11.

Figure 17-2

HA Parameter Timers and AVM Monitoring Example

Gateway A (Active)

AVM11 to AVM99 AVM11 Pings every 60 sec Waits for response 5 min Wait-timer is adjustable (0-15)

Gateway B (Standby)

AVM66 Protection Group A/B

Unit A (Active)

Unit B (Backup)
AVM99 to AVM11 Pings every 60 sec Waits for response 5 min Adjustments to wait-timer are not recommended

AVM101

AVM99 to AVM101 Pings every 60 sec Waits for response 5 min Adjustments to wait-timer are not recommended
AVM99 to AVM100 Pings every 60 sec Waits for response 5 min Adjustments to wait-timer are not recommended

AVM100

Measuring Fault-Processing Down Time for AVMs
When a failure occurs on an AVM, the time during which ticket processing is down is measured as the sum of the following factors:
The time it takes to determine that the AVM has failed. The time it takes for the AVM to reload, depending on the number of VNEs. The time it takes to pass syslogs or traps to the VNEs (in the case of AVM 100), or to pass events to the gateway (in the case of AVM 101-999).
For the first 30 minutes after AVM 99 (the management AVM) has started, there is no monitoring of the system to find unit high availability issues. This allows the system enough time to get up and running.
Impact of Unit Timeouts and Switchovers
The Cisco ANA gateway constantly monitors units by sending a watchdog protocol pulse message to the unit management AVM at preconfigured intervals. If the unit management AVM fails to respond to the pulse message after a preconfigured number of retries, the gateway loads the standby unit to replace it. The impact of such a failure on the system is that the unresponsive unit does not manage the devices for a period of time. This unmanaged period of time is measured by the pulse interval multiplied by the number of retry times, plus the unit load time.

236788

Chapter 17 Configuring Cisco ANA Units for High Availability
Unit load time depends on the configuration of the unitthe hardware, the number of VNEs, the types of VNEs, and the services running on the VNEs. All of these factors impact the load time required for the VNEs to complete their modeling, as described in Table 17-1.
Measuring Ticket-Processing Down Time for Units
When a failure occurs on a unit, the time during which ticket processing is down is measured as the sum of the following factors:
The time it takes to determine that the unit has failed (depending on the ping interval). The time it takes for the unit to reload, depending on the number of AVMs and VNEs in the unit. The time it takes to pass correlated events to the gateway (a minimum of five minutes to obtain device history, plus a variable time depending on the number of VNEs per AVM).
Configuring Cisco ANA Units for High Availability
The following topics describe customizing protection groups, configuring units for high availability, and configuring standby units:
Configuring Units for High Availability Using Protection Groups, page 17-8 Configuring Standby Units, page 17-9 Checking the Assignment of Units to Protection Groups, page 17-10 Changing the Protection Group of a Unit, page 17-10 Switching to a Standby Unit, page 17-11
Configuring Units for High Availability Using Protection Groups
You can change the default settings of a unit and assign it to a customized protection group. For more information about creating, viewing, and deleting protection groups, see Managing Protection Groups, page 6-4. In addition, you can enable or disable high availability for a unit. In other words, these settings enable you to define to which protection group a unit is assigned and whether it is enabled for high availability.
By default, all units in the Cisco ANA fabric belong to the default-pg protection group and high availability is enabled. We recommend that you configure protection groups so that each cluster has two standby units. Advanced configurations can be found in the registry to:

Enable or disable the watchdog protocol for each process, including timeouts for discovery when the process is down. Control the timeouts for detecting when a unit is down.
For more information, see High Availability Registry Settings, page 17-13.
Unit Server High Availability and AVM Protection Configuring Cisco ANA Units for High Availability
To configure a unit for high availability and assign it to a protection group:

Step 1 Step 2

Open the New ANA Unit dialog box by right-clicking the ANA Servers branch, then choose New ANA Unit. Enter the information for the new unit: Field IP Address Description Enter the IP address of the unit. The IP address must be unique.
An error message is displayed if a unit is already configured with the same IP address.
Enable Unit Protection Standby Unit Protection Group
Confirm that this check box is checked. When this option is checked, high availability is enabled on the unit. Confirm that this check box is not checked. Select the protection group for which the newly created unit will act as a standby unit. For more information about protection groups, see:
Managing Protection Groups, page 6-4 Changing the Protection Group of a Unit, page 17-10

Gateway IP

Step 3
Confirm that the IP address of the gateway appears.
Click OK. The new unit is displayed in the Cisco ANA Manage window.
If the new unit is installed and reachable, the following events occur:
It starts automatically. It is registered with the gateway. A configuration registry for the new unit is created in the Golden Source.
Configuring Standby Units
Cisco ANA Manage enables you to configure standby units and assign standby units to protection groups.

Before You Begin

If you are changing an active unit into a standby unit, you must first delete the active unit as described in Connecting and Disconnecting a Cisco ANA Unit, page 3-5. To configure a standby unit:

Step 1

Open the New ANA Unit dialog box by right-clicking the ANA Servers branch, then choose New ANA Unit.
For a detailed description of configuring units, see Chapter 3, Basic Unit Server Administration Tasks.

Step 2

Enter the information for the standby unit: Field IP Address Description Enter the IP address of the unit. The IP address must be unique.

Enable Unit Protection

Confirm that this check box is checked. When this option is checked, high availability is enabled on the unit.
The Enable Unit Protection check box is selected by default. We strongly recommend that you do not disable this option.

Standby Unit Protection Group
Check this check box to define the unit as a standby unit. Select the protection group for which the newly created unit will act as a standby unit. For more information about protection groups, see:

Click OK.

Standby units are not displayed in the ANA Servers branch in the navigation tree.
For information about changing the protection group to which a unit is assigned, see Changing the Protection Group of a Unit, page 17-10.
Checking the Assignment of Units to Protection Groups
You can view the protection groups to which units are currently assigned, allowing you to confirm at a glance that the configuration or assignment matches the initial deployment plan. To view unit and protection group assignments, select the ANA Servers branch in the Cisco ANA Manage navigation pane. The properties of the ANA Servers are displayed in the content area, including the details of the protection group to which each unit and standby unit currently belongs.
Changing the Protection Group of a Unit
You can easily and quickly change the protection group to which a unit has been assigned. To change the protection group of a unit:
Expand the ANA Servers branch and select the required unit. Open the ANA Unit Properties dialog box by right-clicking the unit, then choose Properties.
For a detailed description on configuring units, see Chapter 3, Basic Unit Server Administration Tasks.

Step 3 Step 4

In the Protection Group drop-down list, select the protection group to which you want to assign the unit. Click OK to save the updated protection group setting for the selected unit.
Switching to a Standby Unit
Cisco ANA Manage enables you to switch to a standby unit either manually or automatically.
Automatic switchover to a standby unit occurs when the gateway discovers that one of the active units has failed. Such failures include hardware failures, operating system failures, power failures, and network failures, which disconnect a unit from the Cisco ANA fabric. For more information on automatic switchover, see Unit N+m High Availability, page 17-2. Manually switching to a standby unit is useful if you must temporarily shut down the unit for maintenance.
When a switchover occurs, Cisco ANA automatically transfers all data from the failed unit to a standby unit in the same protection group. The original unit is removed from the standby setup and is no longer displayed in Cisco ANA Manage.
When a unit switches to its standby, all VNEs on the unit that were in maintenance mode will be moved to the VNE Down state. To manually switch to a standby unit:

Step 1 Step 2 Step 3

Expand the ANA Servers branch and select the required unit. Right-click the required unit, then choose Switch. A confirmation message is displayed. Click Yes. The standby unit becomes the active unit and is displayed in the ANA Servers branch. The original unit is removed from the setup and can be safely shut down. It is no longer displayed in the Cisco ANA Manage window.

In the event of unit failover, the Cisco ANA gateway randomly selects a redundant unit when more than one standby unit is available.
Chapter 17 Managing the Watchdog Protocol
Managing the Watchdog Protocol
The following topics describe how to define AVMs for units and enable or disable protection (the watchdog protocol) on the AVM:
Enabling AVM Protection (Watchdog Protocol) on AVMs, page 17-12 Viewing and Changing AVM Protection (Watchdog Protocol) Settings, page 17-13
Enabling AVM Protection (Watchdog Protocol) on AVMs
Every AVM in the Cisco ANA fabric is, by default, managed by the watchdog protocol. Cisco ANA Manage enables you to define AVMs for units and enable or disable the watchdog protocol on each AVM. To define an AVM:
The unit must be installed. The unit must be connected to the transport network. The following default AVMs must be running:
AVM 0The switch AVM. AVM 99The management AVM. AVM 100The trap management AVM (one instance must be running either on the gateway
server or one of the units).
The new AVM must have a unique identifier within the unit.
For detailed information on defining AVMs, see Creating AVMs, Viewing, and Editing AVM Properties, page 4-4. To enable AVM protection on an AVM:
Open the New AVM dialog box by right-clicking the required unit (or gateway), then choose New AVM. Define the properties of the AVM. For more information, see Creating AVMs, Viewing, and Editing AVM Properties, page 4-4. Check the Enable AVM Protection check box to enable the watchdog protocol.

Note Step 4

We strongly recommended that you do not uncheck the Enable AVM Protection check box.
Click OK. The new AVM, with the watchdog protocol enabled, is added to the selected unit and is displayed in the content area. Adding the new AVM creates the registry information for the new AVM in the specified unit. The AVM can now host VNEs.
Unit Server High Availability and AVM Protection High Availability Registry Settings
Viewing and Changing AVM Protection (Watchdog Protocol) Settings
For detailed information on defining and editing AVMs, see Chapter 4, Basic AVM and VNE Administration Tasks. To view and edit AVM settings:
Open the AVM Properties dialog box by right- clicking the required AVM, then choose Properties. Edit the details of the AVM, as required.

Note Step 3

Click OK. The new properties for the AVM are displayed in the content area.
High Availability Registry Settings
The high availability and AVM watchdog protocol functions are controlled by settings in the registry. The registry entries and default values are provided in Table 17-2.
All changes to the registry should only be carried out with the support of Cisco. For details, contact your Cisco account representative.

Table 17-2

Registry Settings for Unit High Availability and AVM Watchdog Protocol

Registry Entry agent_defaults/delay

Description

Default Value
Grace period (in milliseconds) during which events are not 1800000 raised. The grace period begins at system startup. It defines (30 minutes) the amount of time during which the system does not perform high availability operations of any kind on the configured target (either the AVM or the unit). There is one exception: When the configured target responds for the first time with a ping, the grace period is over. Timeout (in milliseconds) for AVMs. This is the initial recovery period. This period includes device polling and inventory buildup. End-to-end services, such as RCA and topology, can take longer before they become available. Timeout (in milliseconds) for units. Threshold (in milliseconds) for AVM reload retries. When exceeded, the AVM is suspended. Maximum number of retries for AVM reloads. When exceeded, the AVM is suspended. 300000 (5 minutes)

agent_defaults/timeout

haservice/timeout agent_defaults/maxTimeoutReloadTime agent_defaults/maxTimeoutReloadTries
300000 (5 minutes) 1800000 (180 minutes) 5
Chapter 17 High Availability Registry Settings