285 lines
9.2 KiB
Plaintext
285 lines
9.2 KiB
Plaintext
|
|
The intent of this file is to give a brief summary of hugetlbpage support in
|
|
the Linux kernel. This support is built on top of multiple page size support
|
|
that is provided by most modern architectures. For example, i386
|
|
architecture supports 4K and 4M (2M in PAE mode) page sizes, ia64
|
|
architecture supports multiple page sizes 4K, 8K, 64K, 256K, 1M, 4M, 16M,
|
|
256M and ppc64 supports 4K and 16M. A TLB is a cache of virtual-to-physical
|
|
translations. Typically this is a very scarce resource on processor.
|
|
Operating systems try to make best use of limited number of TLB resources.
|
|
This optimization is more critical now as bigger and bigger physical memories
|
|
(several GBs) are more readily available.
|
|
|
|
Users can use the huge page support in Linux kernel by either using the mmap
|
|
system call or standard SYSv shared memory system calls (shmget, shmat).
|
|
|
|
First the Linux kernel needs to be built with CONFIG_HUGETLB_PAGE (present
|
|
under Processor types and feature) and CONFIG_HUGETLBFS (present under file
|
|
system option on config menu) config options.
|
|
|
|
The kernel built with hugepage support should show the number of configured
|
|
hugepages in the system by running the "cat /proc/meminfo" command.
|
|
|
|
/proc/meminfo also provides information about the total number of hugetlb
|
|
pages configured in the kernel. It also displays information about the
|
|
number of free hugetlb pages at any time. It also displays information about
|
|
the configured hugepage size - this is needed for generating the proper
|
|
alignment and size of the arguments to the above system calls.
|
|
|
|
The output of "cat /proc/meminfo" will have output like:
|
|
|
|
.....
|
|
HugePages_Total: xxx
|
|
HugePages_Free: yyy
|
|
Hugepagesize: zzz KB
|
|
|
|
/proc/filesystems should also show a filesystem of type "hugetlbfs" configured
|
|
in the kernel.
|
|
|
|
/proc/sys/vm/nr_hugepages indicates the current number of configured hugetlb
|
|
pages in the kernel. Super user can dynamically request more (or free some
|
|
pre-configured) hugepages.
|
|
The allocation( or deallocation) of hugetlb pages is posible only if there are
|
|
enough physically contiguous free pages in system (freeing of hugepages is
|
|
possible only if there are enough hugetlb pages free that can be transfered
|
|
back to regular memory pool).
|
|
|
|
Pages that are used as hugetlb pages are reserved inside the kernel and can
|
|
not be used for other purposes.
|
|
|
|
Once the kernel with Hugetlb page support is built and running, a user can
|
|
use either the mmap system call or shared memory system calls to start using
|
|
the huge pages. It is required that the system administrator preallocate
|
|
enough memory for huge page purposes.
|
|
|
|
Use the following command to dynamically allocate/deallocate hugepages:
|
|
|
|
echo 20 > /proc/sys/vm/nr_hugepages
|
|
|
|
This command will try to configure 20 hugepages in the system. The success
|
|
or failure of allocation depends on the amount of physically contiguous
|
|
memory that is preset in system at this time. System administrators may want
|
|
to put this command in one of the local rc init file. This will enable the
|
|
kernel to request huge pages early in the boot process (when the possibility
|
|
of getting physical contiguous pages is still very high).
|
|
|
|
If the user applications are going to request hugepages using mmap system
|
|
call, then it is required that system administrator mount a file system of
|
|
type hugetlbfs:
|
|
|
|
mount none /mnt/huge -t hugetlbfs <uid=value> <gid=value> <mode=value>
|
|
<size=value> <nr_inodes=value>
|
|
|
|
This command mounts a (pseudo) filesystem of type hugetlbfs on the directory
|
|
/mnt/huge. Any files created on /mnt/huge uses hugepages. The uid and gid
|
|
options sets the owner and group of the root of the file system. By default
|
|
the uid and gid of the current process are taken. The mode option sets the
|
|
mode of root of file system to value & 0777. This value is given in octal.
|
|
By default the value 0755 is picked. The size option sets the maximum value of
|
|
memory (huge pages) allowed for that filesystem (/mnt/huge). The size is
|
|
rounded down to HPAGE_SIZE. The option nr_inode sets the maximum number of
|
|
inodes that /mnt/huge can use. If the size or nr_inode options are not
|
|
provided on command line then no limits are set. For size and nr_inodes
|
|
options, you can use [G|g]/[M|m]/[K|k] to represent giga/mega/kilo. For
|
|
example, size=2K has the same meaning as size=2048. An example is given at
|
|
the end of this document.
|
|
|
|
read and write system calls are not supported on files that reside on hugetlb
|
|
file systems.
|
|
|
|
A regular chown, chgrp and chmod commands (with right permissions) could be
|
|
used to change the file attributes on hugetlbfs.
|
|
|
|
Also, it is important to note that no such mount command is required if the
|
|
applications are going to use only shmat/shmget system calls. Users who
|
|
wish to use hugetlb page via shared memory segment should be a member of
|
|
a supplementary group and system admin needs to configure that gid into
|
|
/proc/sys/vm/hugetlb_shm_group. It is possible for same or different
|
|
applications to use any combination of mmaps and shm* calls. Though the
|
|
mount of filesystem will be required for using mmaps.
|
|
|
|
*******************************************************************
|
|
|
|
/*
|
|
* Example of using hugepage memory in a user application using Sys V shared
|
|
* memory system calls. In this example the app is requesting 256MB of
|
|
* memory that is backed by huge pages. The application uses the flag
|
|
* SHM_HUGETLB in the shmget system call to inform the kernel that it is
|
|
* requesting hugepages.
|
|
*
|
|
* For the ia64 architecture, the Linux kernel reserves Region number 4 for
|
|
* hugepages. That means the addresses starting with 0x800000... will need
|
|
* to be specified. Specifying a fixed address is not required on ppc64,
|
|
* i386 or x86_64.
|
|
*
|
|
* Note: The default shared memory limit is quite low on many kernels,
|
|
* you may need to increase it via:
|
|
*
|
|
* echo 268435456 > /proc/sys/kernel/shmmax
|
|
*
|
|
* This will increase the maximum size per shared memory segment to 256MB.
|
|
* The other limit that you will hit eventually is shmall which is the
|
|
* total amount of shared memory in pages. To set it to 16GB on a system
|
|
* with a 4kB pagesize do:
|
|
*
|
|
* echo 4194304 > /proc/sys/kernel/shmall
|
|
*/
|
|
#include <stdlib.h>
|
|
#include <stdio.h>
|
|
#include <sys/types.h>
|
|
#include <sys/ipc.h>
|
|
#include <sys/shm.h>
|
|
#include <sys/mman.h>
|
|
|
|
#ifndef SHM_HUGETLB
|
|
#define SHM_HUGETLB 04000
|
|
#endif
|
|
|
|
#define LENGTH (256UL*1024*1024)
|
|
|
|
#define dprintf(x) printf(x)
|
|
|
|
/* Only ia64 requires this */
|
|
#ifdef __ia64__
|
|
#define ADDR (void *)(0x8000000000000000UL)
|
|
#define SHMAT_FLAGS (SHM_RND)
|
|
#else
|
|
#define ADDR (void *)(0x0UL)
|
|
#define SHMAT_FLAGS (0)
|
|
#endif
|
|
|
|
int main(void)
|
|
{
|
|
int shmid;
|
|
unsigned long i;
|
|
char *shmaddr;
|
|
|
|
if ((shmid = shmget(2, LENGTH,
|
|
SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W)) < 0) {
|
|
perror("shmget");
|
|
exit(1);
|
|
}
|
|
printf("shmid: 0x%x\n", shmid);
|
|
|
|
shmaddr = shmat(shmid, ADDR, SHMAT_FLAGS);
|
|
if (shmaddr == (char *)-1) {
|
|
perror("Shared memory attach failure");
|
|
shmctl(shmid, IPC_RMID, NULL);
|
|
exit(2);
|
|
}
|
|
printf("shmaddr: %p\n", shmaddr);
|
|
|
|
dprintf("Starting the writes:\n");
|
|
for (i = 0; i < LENGTH; i++) {
|
|
shmaddr[i] = (char)(i);
|
|
if (!(i % (1024 * 1024)))
|
|
dprintf(".");
|
|
}
|
|
dprintf("\n");
|
|
|
|
dprintf("Starting the Check...");
|
|
for (i = 0; i < LENGTH; i++)
|
|
if (shmaddr[i] != (char)i)
|
|
printf("\nIndex %lu mismatched\n", i);
|
|
dprintf("Done.\n");
|
|
|
|
if (shmdt((const void *)shmaddr) != 0) {
|
|
perror("Detach failure");
|
|
shmctl(shmid, IPC_RMID, NULL);
|
|
exit(3);
|
|
}
|
|
|
|
shmctl(shmid, IPC_RMID, NULL);
|
|
|
|
return 0;
|
|
}
|
|
|
|
*******************************************************************
|
|
|
|
/*
|
|
* Example of using hugepage memory in a user application using the mmap
|
|
* system call. Before running this application, make sure that the
|
|
* administrator has mounted the hugetlbfs filesystem (on some directory
|
|
* like /mnt) using the command mount -t hugetlbfs nodev /mnt. In this
|
|
* example, the app is requesting memory of size 256MB that is backed by
|
|
* huge pages.
|
|
*
|
|
* For ia64 architecture, Linux kernel reserves Region number 4 for hugepages.
|
|
* That means the addresses starting with 0x800000... will need to be
|
|
* specified. Specifying a fixed address is not required on ppc64, i386
|
|
* or x86_64.
|
|
*/
|
|
#include <stdlib.h>
|
|
#include <stdio.h>
|
|
#include <unistd.h>
|
|
#include <sys/mman.h>
|
|
#include <fcntl.h>
|
|
|
|
#define FILE_NAME "/mnt/hugepagefile"
|
|
#define LENGTH (256UL*1024*1024)
|
|
#define PROTECTION (PROT_READ | PROT_WRITE)
|
|
|
|
/* Only ia64 requires this */
|
|
#ifdef __ia64__
|
|
#define ADDR (void *)(0x8000000000000000UL)
|
|
#define FLAGS (MAP_SHARED | MAP_FIXED)
|
|
#else
|
|
#define ADDR (void *)(0x0UL)
|
|
#define FLAGS (MAP_SHARED)
|
|
#endif
|
|
|
|
void check_bytes(char *addr)
|
|
{
|
|
printf("First hex is %x\n", *((unsigned int *)addr));
|
|
}
|
|
|
|
void write_bytes(char *addr)
|
|
{
|
|
unsigned long i;
|
|
|
|
for (i = 0; i < LENGTH; i++)
|
|
*(addr + i) = (char)i;
|
|
}
|
|
|
|
void read_bytes(char *addr)
|
|
{
|
|
unsigned long i;
|
|
|
|
check_bytes(addr);
|
|
for (i = 0; i < LENGTH; i++)
|
|
if (*(addr + i) != (char)i) {
|
|
printf("Mismatch at %lu\n", i);
|
|
break;
|
|
}
|
|
}
|
|
|
|
int main(void)
|
|
{
|
|
void *addr;
|
|
int fd;
|
|
|
|
fd = open(FILE_NAME, O_CREAT | O_RDWR, 0755);
|
|
if (fd < 0) {
|
|
perror("Open failed");
|
|
exit(1);
|
|
}
|
|
|
|
addr = mmap(ADDR, LENGTH, PROTECTION, FLAGS, fd, 0);
|
|
if (addr == MAP_FAILED) {
|
|
perror("mmap");
|
|
unlink(FILE_NAME);
|
|
exit(1);
|
|
}
|
|
|
|
printf("Returned address is %p\n", addr);
|
|
check_bytes(addr);
|
|
write_bytes(addr);
|
|
read_bytes(addr);
|
|
|
|
munmap(addr, LENGTH);
|
|
close(fd);
|
|
unlink(FILE_NAME);
|
|
|
|
return 0;
|
|
}
|