5 Setting Up the MEMORY CHANNEL Cluster Interconnect

This chapter describes MEMORY CHANNEL configuration requirements, and describes how to set up the MEMORY CHANNEL cluster interconnect, including setting up a MEMORY CHANNEL hub and connecting link cables.

If you are setting up a TruCluster Available Server Software configuration, you can ignore this chapter as MEMORY CHANNEL hardware is not used.

Section 2.4 lists the supported MEMORY CHANNEL hardware. See the MEMORY CHANNEL User's Guide for illustrations and detailed information about installing jumpers, MEMORY CHANNEL adapters, and hubs. See Chapter 7 for information about changing and maintaining the MEMORY CHANNEL hardware.

TruCluster Production Server Software and TruCluster MEMORY CHANNEL Software currently support only MEMORY CHANNEL for cluster interconnects. You can have two MEMORY CHANNEL adapters for TruCluster Production Server Software to protect against an interconnect failure and for easier hardware maintenance. If the active MEMORY CHANNEL adapter fails, cluster communications fails over to the inactive MEMORY CHANNEL adapter.

You can have up to four MEMORY CHANNEL adapters for TruCluster MEMORY CHANNEL Software but there is no failover between adapters. For each interconnect, you must install one MEMORY CHANNEL adapter in a peripheral component interconnect (PCI) slot in each member system.

See Section 5.1 for a discussion on MEMORY CHANNEL requirements.

To set up the MEMORY CHANNEL interconnects, follow these steps, referring to the appropriate section and the MEMORY CHANNEL User's Guide as necessary:

Set the MEMORY CHANNEL jumpers (Section 5.2).

Install the MEMORY CHANNEL adapter into a PCI slot on each system (Section 5.3).

If you have more than two systems in the cluster, install a MEMORY CHANNEL hub (Section 5.4).

Connect the MEMORY CHANNEL cables (Section 5.5).

After you complete steps 1 through 4 for all systems in the cluster, apply power to the systems and run MEMORY CHANNEL diagnostics (Section 5.6).

Note

If you are installing SCSI or network adapters, you may wish to complete all hardware installation before powering up the systems to run MEMORY CHANNEL diagnostics.

5.1 MEMORY CHANNEL Requirements

Ensure that you abide by the following MEMORY CHANNEL requirements:

MEMORY CHANNEL adapters require a specific, minimum firmware revision. See the TruCluster Software Products Release Notes for information about required firmware revisions.

For AlphaServer 2000 systems, the B2111-AA module must be at Revision H or higher.
For AlphaServer 2100 systems, the B2110-AA module must be at Revision L or higher.
Use the examine console command to determine if these modules are at a supported revision as follows:
```
P00>>> examine -b econfig:20008
econfig: 20008 04
P00>>>
```
If a hexadecimal value of 04 or greater is returned, the I/O module supports MEMORY CHANNEL.
If a hexadecimal value of less than 04 is returned, the I/O module is not supported for MEMORY CHANNEL usage.
Order an H3095-AA module to upgrade an AlphaServer 2000 or an H3096-AA module to upgrade an AlphaServer 2100 to support MEMORY CHANNEL.

For AlphaServer 1000A systems, the MEMORY CHANNEL adapter must be installed on the primary PCI (in front of the PCI-to-PCI bridge chip) in PCI slots 11, 12, or 13 (the top three slots).

For AlphaServer 8200/8400 systems, the MEMORY CHANNEL adapter must be installed in slots 0-7 of a DWLPA-CA PCI; there are no restrictions for a DWLPB-CA PCI bus.

A MEMORY CHANNEL interconnect can use either virtual hub mode (two member systems connected without a MEMORY CHANNEL hub) or standard mode (two or more systems connected to a MEMORY CHANNEL hub).

If you use multiple MEMORY CHANNEL interconnects, each interconnect must use the same mode, either virtual hub mode or standard mode.

If you are using virtual hub mode, you must connect the first MEMORY CHANNEL adapter installed in one system to the first adapter installed in the other system, and you must connect the second adapter installed in one system to the second adapter installed in the other system.
If a Production Server or MEMORY CHANNEL configuration utilizes multiple MEMORY CHANNEL adapters in standard hub mode, the MEMORY CHANNEL adapters must be connected to separate MEMORY CHANNEL hubs. The first MEMORY CHANNEL adapter in each system must be connected to one MEMORY CHANNEL hub. The second MEMORY CHANNEL adapter in each system must be connected to a second MEMORY CHANNEL hub.

The maximum length of a MEMORY CHANNEL link cable is 3 meters(10 feet).The maximum length of a MEMORY CHANNEL 2 link cable is 10 meters.
Always check a link cable for bent or broken pins. Be sure that you do not bend or break any pins when you connect or disconnect a cable.

5.2 Setting the MEMORY CHANNEL Adapter Jumpers

The MEMORY CHANNEL module has adapter jumpers that designate whether the configuration is using standard or virtual hub mode. If virtual hub mode is being used, there can be only two systems. One system must be virtual hub 0 (VH0) and the other must be virtual hub 1 (VH1).

The MEMORY CHANNEL adapter should arrive with the jumpers set for standard hub mode (jumpering middle and left pins). Confirm that the jumpers are set properly. The jumpers are right next to the factory/maintenance cable connector and are shown in Table 5-1.

Table 5-1: MEMORY CHANNEL Jumper Configuration

If mode is: Jumper: Example:

Standard Middle and left pins

Virtual: VH0 Middle and right pins

Virtual: VH1 None needed; store the jumper on the right post

If mode is:	Jumper:	Example:
Standard	Middle and left pins
Virtual: VH0	Middle and right pins
Virtual: VH1	None needed; store the jumper on the right post

If you are upgrading from virtual hub mode to standard mode (or from standard hub mode to virtual hub mode), be sure to change the jumpers on all MEMORY CHANNEL adapters.

5.3 Installing the MEMORY CHANNEL Adapter

Install the MEMORY CHANNEL adapter in an appropriate peripheral component interconnect (PCI) slot. Secure the module at the backplane. Ensure that the screw is tight to maintain proper grounding, but do not over torque the screw. An over-torqued screw can pull the module partially out of the backplane connector.

The MEMORY CHANNEL adapter comes with a straight extension plate. This fits most systems; however, you may have to replace the extender with an angled extender, or for an AlphaServer 8200/8400, remove the extender completely.

If you are setting up a redundant MEMORY CHANNEL configuration, install the second MEMORY CHANNEL adapter right after installing the first MEMORY CHANNEL adapter. Ensure that the jumpers are correct and are the same on both modules.

After you install the MEMORY CHANNEL adapter(s), replace the system panels.

5.4 Installing the MEMORY CHANNEL Hub

You may use a hub in a two-node MEMORY CHANNEL cluster, but the hub is not required. When there are more than two systems in a MEMORY CHANNEL cluster, you must use a MEMORY CHANNEL hub as follows:

You must install the hub within 3 meters (10 feet) of each of the systems.

If the hub has a voltage selection switch on the back of the hub, select the correct voltage for your location.

Ensure that the hub contains a CCMLA line card for each system in the cluster (the hub comes with four line cards). If you have a four-node cluster, you may want to install an extra line card for troubleshooting use.

5.5 Installing the MEMORY CHANNEL Cables

MEMORY CHANNEL cable installation depends upon the MEMORY CHANNEL module revision, and whether or not you are using fiber optics.

5.5.1 Installing the MEMORY CHANNEL Cables

To set up a MEMORY CHANNEL interconnect, use the BC12N-10 MEMORY CHANNEL link cables to connect MEMORY CHANNEL adapters and, optionally, MEMORY CHANNEL hubs. The maximum length of the link cable is 3 meters (10 feet).

5.5.2 Connecting MEMORY CHANNEL Link Cables in Virtual Hub Mode

Link Cables in Virtual Hub Mode

For a MEMORY CHANNEL virtual hub configuration (two nodes in the cluster), connect the BC12N-10 link cables between the MEMORY CHANNEL adapter installed in each of the systems.

Caution

Be very careful when installing the link cables. Insert the cables straight in. When removing the cables, rock them vertically, not horizontally.

Gently push the cables' connector into the receptacle, then use the screws to pull the connector in tight. The connector must be tight to ensure a good ground contact.

If you are setting up redundant interconnects:

Both adapters in a system must have the same jumper setting, either VH0 or VH1.

You must connect the first MEMORY CHANNEL adapter installed in one system to the first adapter installed in the other system, and connect the second adapter in one system to the second adapter in the other system.

5.5.3 Connecting MEMORY CHANNEL Link Cables in Standard Hub Mode

If there are more than two systems in a cluster, use a MEMORY CHANNEL standard hub configuration. Connect a BC12N-10 link cable between the MEMORY CHANNEL adapter and a line card in the CCMHA-AA hub, starting at the lowest numbered slot in the hub.

If you are setting up redundant interconnects, the following rules apply:

Each adapter installed in a system must be connected to a different hub.

Each MEMORY CHANNEL adapter in a system must be connected to line cards that are installed in the same slot position in each hub. For example, if you connect one adapter to a line card installed in slot one in one hub, you must connect the other adapter in that system to a line card installed in slot one of the second hub.

Figure 5-1 shows MEMORY CHANNEL adapters connected to line cards that are in the same slot position in the MEMORY CHANNEL hubs.

Figure 5-1: Connecting MEMORY CHANNEL Adapters to Hubs

5.6 Running MEMORY CHANNEL Diagnostics

After the MEMORY CHANNEL adapters, hubs, and link cables have been installed, power up the systems and run the MEMORY CHANNEL diagnostics.

There are two console level MEMORY CHANNEL diagnostics, mc_diag and mc_cable:

The mc_diag diagnostic:
- Tests the MEMORY CHANNEL adapter(s) on the system running the diagnostic.
- It is run as part of the initialization sequence when the system is powered up.
- Can be run on a standalone system or while connected to another system or a hub with the link cable.

The mc_cable diagnostic:
- Must be run on all systems in the cluster simultaneously (therefore, all systems must be at the console prompt).
- Is designed to isolate problems to the MEMORY CHANNEL adapter, BC12N-10 link cables, hub line cards, and, to some extent, to the hub.
- Indicates data flow through the MEMORY CHANNEL by response messages.
- Runs continuously until terminated with Ctrl/C.
- Reports differences in connection state, not errors.
- Can be run in standard or virtual hub mode.

When the console indicates a successful response from all other systems being tested, the data flow through the MEMORY CHANNEL hardware has been completed and the test may be terminated by pressing Ctrl/C on each system being tested.

Caution

Do not run the mc_cable MEMORY CHANNEL diagnostic on a system that is cabled to a live cluster. The diagnostic transmissions on the link are seen by other systems in the cluster and may cause problems.

Example 5-1 shows a sample output from one node of a standard hub configuration. In this example, the test is started on node 1, then on node 0. The test must be terminated on each system.

Example 5-1: Running the `mc_cable` Test

>>> mc_cable                       [1]
To exit MC_CABLE, type <Ctrl/C>
mca0 node id 1 is online           [2]
No response from node 0 on mca0    [2]
mcb0 node id 1 is online           [3]
No response from node 0 on mcb0    [3]
Response from node 0 on mca0       [4]
Response from node 0 on mcb0       [5]
mcb0 is offline                    [6]
mca0 is offline                    [6]
[Ctrl/C]                              [7]
>>>

The mc_cable diagnostic is initiated on node 1. [Return to example]

Node one reports that mca0 is on line but has not communicated with the MEMORY CHANNEL adapter on node 0. [Return to example]

Node one reports that mcb0 is on line but has not communicated with the MEMORY CHANNEL adapter on node 0. [Return to example]

MEMORY CHANNEL adapter mca0 has communicated with the adapter on the other node. [Return to example]

MEMORY CHANNEL adapter mcb0 has communicated with the adapter on the other node. [Return to example]

Typing a Ctrl/C on node 0 terminates the test on that node and the MEMORY CHANNEL adapters on node one report off line. [Return to example]

A Ctrl/C on node one terminates the test. [Return to example]