Q.) I received this question from a partner in a ProLiant course and was inquiring if anyone could provide some feedback?
What impact will enabling the ?Node Interleaving? option in the BIOS of HP servers have on the OS and other applications, particularly as it relates to both Windows Server 2003 and 2008. How does the option affect NUMA aware OS (excludes Windows Server 2003 Standard Edition)? I though if ?Node Interleaving? is enabled in BIOS is actually means the server is running UMA (Uniform Memory Architecture).
A.) This option was first added into the DL585 rbsu, the first NUMA x86 server that HP launched. When node interleave is disabled (default setting), the processor proceeds memory page r/w on its own node first, and once its local memory is full, then proceeds with memory on remote node (closest node). When node interleave set to enabled, the processor read/write pages equally on local and remote memory. As most of x86 apps are NOT numa aware, the setting is disabled by default. This is not the case for oracle db app.
Starting with G6 and nehalem, intel x86 servers are NUMA ready.
Here?s a relatively old SAW article that explains how it works.
HP ProLiant DL585 Server Series – What Is Node Interleaving?
AMD-based HP ProLiant DL585 series of servers provide two methods of organizing memory access:
- linear, non-uniform memory access (NUMA)
- node interleaving, sufficiently uniform memory accessing (SUMA)
Node:
A node consists of one or more processors, its embedded memory controller, and the attached DIMMs. The total memory attached to all the processors is divided into 4-KB segments.
In case of linear addressing (NUMA), consecutive 4-KB segments are on the same node.
In case of node interleaving (SUMA), consecutive 4-KB segments are on different or adjacent nodes.
Linear memory accessing or NUMA
Linear memory accessing or NUMA defines the memory on all nodes sequentially. It assigns sequential addresses to all memory locations on node 0, then to all the memory locations on node 1, and so on until memory locations on all nodes have been assigned.
Node interleaving or SUMA
Node interleaving or SUMA breaks memory into 4-KB addressable entities. Addressing starts with address 0 on node 0. Sequential addresses through address 4095 are assigned to node 0, addresses 4096 through 8191 to node 1, addresses 8192 through 12287 to node 2, and addresses 12888 through 16383 to node 3. Address 16384 is assigned to node 0, and the process continues until all memory has been assigned in this fashion.
Applications that do not benefit from node interleaving
To take advantage of NUMA architecture, the operating system and the applications that run on the system must assign memory on a per-thread basis. In general, a NUMA-aware operating system such as Microsoft Windows and a NUMA-aware application such as Microsoft SQL Server will benefit from the NUMA organization. A NUMA-aware operating system and applications that allocate and de-allocate memory at the thread level will benefit from the NUMA organization because the allocation and the thread will have a tendency to run on the same node.
Applications that benefit from node Interleaving:
If an application uses a common allocation thread, it will benefit from node interleaving.