shmget: Not enough space What does this error mean? If you're lucky enough to actually get this error message, troubleshooting the cause, and finding the resolution or suitable workaround, should be relatively straightforward. On the other hand, the database or other application you're running may simply hang, or give some other indication that it can't access some resource on the system. Alternatively, it may tell you that there's not enough memory, or that there is insufficient table space. If you suspect that shared memory allocation is the problem, how can you verify this? Once verified, is there anything you can do to resolve the problem, or at least workaround the problem? This article will explain some of the terms associated with shared memory allocation and discuss the limits for some of the shared memory parameters by operating system version. Additionally, it will provide some handy commands for approximating how much shared memory space is left on the system, and discuss some high-level methods for making efficient use of the shared memory on the system.
Shared Memory Terminology
The first concept to address is the definition of a Shared Memory Segment. Imagine all of the memory on the system as being in one big bucket, and refer to the bucket as the Global Shared Memory Pool. If a chunk of this memory is allocated, this chunk can be thought of as a segment of shared memory. This chunk of shared memory, or shared memory segment, must meet one very important requirement: All of the memory addresses must be allocated in sequence, with no gaps between addresses. Memory allocated in this manner is known as a chunk of contiguous memory.
The second concept is that of the total address space in a PA-RISC system. The total address space of a modern PA-RISC (pre 64-bit HP-UX) system is 4 gigabytes. This space is broken down into four equal size sections called quadrants, each one gigabyte in size. Now for the next very important rule: No shared memory segment may cross a quadrant boundary. The previous statement implies that no single shared memory segment may be larger than one gigabyte. This is true for all versions of HP-UX 9.X and 10.X.
The third concept that this article will discuss is that of Shared Memory Fragmentation. For the purpose of discussion, when allocating shared memory, the kernel will allocate the first contiguous shared memory segment that satisfies the request. Because of allocating memory this way, larger chunks of memory are broken into smaller chunks. Imagine if you will, needing a piece of lumber 2-feet long. Given the choice of taking 2 feet from a 10-foot board, or from a 16-foot board, you'd probably choose the 10-foot board, as this would leave the 16-foot board intact. With shared memory, since we allocate memory from the first segment that can satisfy the request, there's no guarantee that the request was honored from the smallest available chunk of memory. Therefore, it's not surprising to find that a shmget() call can not be satisfied because no available shared memory segment is large enough, even though there may be plenty of shared memory in the shared memory pool. This might happen on a system which has gone a long time without a reboot, or which has many processes.
By now, the pieces of the puzzle should be falling into place.
Global Shared Memory Pool Limits
As was pointed out earlier in this article, the limit to the size of an individual shared memory segment is limited by the size of a quadrant, which is 1 gigabyte; think of this as the hard limit. There is a kernel parameter that will further restrict the size of a shared memory segment: shmmax. If you're concerned with trimming the fat from your kernel, then adjust the value of shmmax. Consider setting shmmax to 10% greater than the largest shared memory segment you expect any application to allocate. The following table summarizes the total amount of shared memory that can be allocated on a specific version of HP-UX.
| HP-UX Version | Total Shared Memory | Comments |
| 9.X | 750 megabytes | -quadrant 4 only -250 megabytes of quadrant 4 for memory-mapped I/O |
| 10.01 | 1.75 gigabytes | -quadrants 3 and 4 |
| 10.10 | 1.75 gigabytes | -quadrants 3 and 4 |
| 10.20 | 2.75 gigabytes | -quadrants 3 and 4 -quadrant 2 with patch |
| 10.30 | 2.75 gigabytes | -quadrants 2, 3, and 4 |
Beginning at HP-UX version 10.20, up to 2.75 gigabytes of shared memory can be allocated, but some special steps need to be taken in order to get access to the memory in quadrant 2. First, the following patches must be installed: PHKL_12669 (s700) or PHKL_12670 (s800), and PHSS_10926. Second, an existing application may be relinked as the new executable type SHMEM_MAGIC, or, the application can be linked as type EXEC_MAGIC, and then chatr'd to be the new executable type SHMEM_MAGIC. The patch PHSS_10926 provides the ability for chatr(1) to make this change. As of the writing of this article, only the chatr(1) method is supported. This implies that it's not possible to link the executable as type SHMEM_MAGIC.
Determining Shared Memory Utilization
So now that you've developed a good baseline understanding of shared memory concepts, how can you put this knowledge to use? Let's start by reviewing the basics:
-There is an upper limit, depending on OS version, to the amount of shared memory that
may be allocated.
-No single shared memory segment may exceed 1 gigabyte.
-A shared memory segment must be comprised of contiguous memory pages
From the previous statements, we see several ways in which our process might not be able to obtain the shared memory it needs:
1) Not enough virtual memory on the system
2) Size of the segment that we are trying to allocate is larger than 1 gigabyte
3) No available, contiguous chunk of memory to satisfy the request
# swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 49152 0 49152 0% 0 - 1 /dev/vg00/lvol2
Now, some printouts from an actual system will be shown. The representative system is a model 735 running HP-UX 10.01. To establish a baseline on this system, swapinfo(1M) will be used to determine how much available swap space exists on this system. Then, the amount of shared memory allocated immediately after bootup will be obtained by using the ipcs(1) command. Figure 2. shows the output of swapinfo -ta:
8260 40892 17% 0 - 0 /dev/vg00/lvol15 dev 40960 8296 32664 20% 0 - 0 /dev/vg00/lvol3 dev 1884160 0 1884160 0% 0 - 1 /dev/vg00/lvbal reserve - 44240 -44240 total 2023424 60796 1962628 3% - 0 -figure 2.
Currently, this system has almost 2 GB of swap available, according to the field total Kb FREE. Therefore, this system will reach its maximum shared memory value of 1.75 GB long before a shortage of swap space becomes a problem. Next, the output of the ipcs(1) command will show how much shared memory is currently allocated on the system. See figure 3:
# ipcs -mob IPC status from /dev/kmem as of Fri Oct 17 00:37:57 1997 T ID KEY MODE OWNER GROUP NATTCH SEGSZ Shared Memory: m 0 0x411c032c --rw-rw-rw- root root 0 348 m 1 0x4e040002 --rw-rw-rw- root root 1 31008 m 2 0x41180037 --rw-rw-rw- root root 1 8192 m 3 0xff1cae0b --rw-rw-rw- root root 2 84 m 4 0x00000000 --rw------- root sys 2 65536 m 5 0xfe1cae0b --rw-rw-rw- root root 2 24652 m 6 0xfd1cae0b --rw-rw-rw- root root 2 1804 m 7 0xfc1cae0b --rw-rw-rw- root root 2 20812 m 8 0x33440186 --rw------- root root 3 1119096 m 9 0x6d440002 --rw-rw-rw- root root 2 47120 m 212 0x431c1b59 --rw-rw-rw- daemon daemon 2 5767168 m 13 0x00000000 D-rw------- daemon daemon 2 524384figure 3.
Totaling up the numbers in the SEGSZ column shows that slightly less than 8 MB of shared memory is allocated on this system, immediately after a reboot. The absolute debugger, adb(1), will now be used to show how much free memory is available in each of the two quadrants where shared memory is available. As this article's intent is to educate the reader regarding shared memory allocation, no attempt will be made to review the use of adb(1), nor will any explanation be given as to where this information was derived from. Adb(1) can be used to examine the largest contiguous chunk of memory available in the quadrant of interest. A step-by-step explanation follows:
1) Start adb(1):
# adb -k /stand/vmunix /dev/mem
2) Enter the command for examining the free memory in quadrant 3:
*quad3map,20/X
5FC64C: 5FD2D4 29530 40000 80001
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
3) Don’t be concerned with the first three values; find the value in the last row, second to last column, which in this case is 40000. This value represents the hexadecimal number of 4 KB contiguous memory pages that are available in this quadrant. The next number, 80001, is the starting address of this memory. Throughout the rest of the adb(1) listing, the same order is followed. Also keep in mind that the last amount/offset pair is usually the largest memory segment in the quadrant, but if memory is badly fragmented, the largest segment might be in the middle of the map. Whichever is the case, find the largest segment. A good way to make sure you're looking at the correct value for the number of pages is to go to the last row, rightmost column; this is the starting address of the amount of memory pages shown in the previous column.
4) Use adb to convert this number to decimal bytes. Note that since the original value was in 4 KB pages, adding three zeros will convert the number to hexadecimal bytes. The "=D" converts the value to decimal.
40000000=D
1073741824
This computation shows that about 1 GB is available in quadrant 3. So, from what we've learned already, if we do the same steps for quadrant 4, we should see somewhere around 750 MB available.
# adb -k /stand/vmunix /dev/mem
*quad4map,20/X
5FD2D4: 5FDF5C 2953C 2E8A9 0xC1758
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
2e8a9000=D
780832768
We're now ready to see what effect starting some processes which allocate shared memory will have on the system. A program that allocates two shared memory segments has been started. Each segment is 700 megabytes in size. First, let's see how swapinfo -ta looks now:
# swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 49152 0 49152 0% 0 - 1 /dev/vg00/lvol2
dev 49152 8776 40376 18% 0 - 0 /dev/vg00/lvol15
dev 40960 9164 31796 22% 0 - 0 /dev/vg00/lvol3
dev 1884160 0 1884160 0% 0 - 1 /dev/vg00/lvbal
reserve - 1482388 -1482388
total 2023424 1500328 523096 74% - 0 -
As one can see, a significant piece of the total swap on the system has been allocated. According to swapinfo(1M), there remains about 500 MB of swap space to be allocated. What does ipcs(1) show now?
# ipcs -mob IPC status from /dev/kmem as of Fri Oct 17 01:46:22 1997 T ID KEY MODE OWNER GROUP NATTCH SEGSZ Shared Memory: m 0 0x411c032c --rw-rw-rw- root root 0 348 m 1 0x4e040002 --rw-rw-rw- root root 1 31008 m 2 0x41180044 --rw-rw-rw- root root 1 8192 m 3 0xff1cae0b --rw-rw-rw- root root 2 84 m 4 0x00000000 --rw------- root sys 2 65536 m 5 0xfe1cae0b --rw-rw-rw- root root 2 24652 m 6 0xfd1cae0b --rw-rw-rw- root root 2 1804 m 7 0xfc1cae0b --rw-rw-rw- root root 2 20812 m 8 0x33440186 --rw------- root root 2 1119096 m 9 0x6d440002 --rw-rw-rw- root root 2 47120 m 210 0x431c1b59 --rw-rw-rw- daemon daemon 2 5767168 m 11 0x00000000 D-rw------- daemon daemon 2 524384 m 12 0x00000010 --rw------- root sys 2 734003200 m 13 0x00000015 --rw------- root sys 2 734003200
The two shared memory segments we've created are easy to spot, and are assigned ID 12 and ID 13. Since these segments are orders of magnitude larger than the other segments, it's safe to say that there is about 1.4 GB of shared memory allocated on the system. Subtracting 1.4 GB from 1.75 GB leaves about 350 MB of shared memory left on the system. This isn't the whole picture though. Remember that ipcs(1) will show the size of the segments being used, but doesn't give any hint as to how fragmented the memory might be, or what the size of the largest contiguous chunk of memory is. This is where adb(1) shines:
# adb -k /stand/vmunix /dev/mem
*quad3map,20/X
5FC64C: 5FD2D4 29530 14400 0xABC01
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
14400000=D
339738624
*quad4map,20/X
5FD2D4: 5FDF5C 2953C 2CA9 0xED358
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
2ca9000=D
46829568
Note that there is about 325 MB available in quadrant 3, but only around 44 MB available in quadrant 4. If a program is run which attempts to allocate two 180 MB shared memory segments, the shmget()will fail This is because 360 MB is more than can be allocated from either quadrant. Since the segment must be contained within one quadrant, there is no way to allocate this memory. Also, keep in mind that if the memory in either quadrant were badly fragmented, this would further reduce the size of the largest contiguous shared memory segment. Occasionally, a system that has hundreds of megabytes of shared memory available not be able to shmget() any segment larger than 20 or 30 MB.
The last point to be made in this section will be to address the issue of cleaning up shared memory that is no longer in use. In regards to cleaning up unused shared memory segments, the application is responsible for ensuring that all unused shared memory segments are removed. For example, the piece of code that was written to create the segments shown in this article was not written to be able to handle errors or interruptions. If the process is killed, the shared memory segments are still left around. Another familiar utility that behaves the same way, for example, if it is terminated with a SIGKILL (kill -9) is fbackup(1M). The fbackup(1M) manpage states that "fbackup allocates resources that are not returned to the system if it is killed in an ungraceful manner". Remember this the next time you kill a process. Never use a SIGKILL (kill -9) unless absolutely necessary. If, on the other hand, the shared memory segment is removed after the process has started, and is still running, the segments will be marked as follows:
# ipcs -mob IPC status from /dev/kmem as of Fri Oct 17 02:16:31 1997 T ID KEY MODE OWNER GROUP NATTCH SEGSZ m 212 0x00000000 D-rw------- root sys 2 734003200 m 213 0x00000000 D-rw------- root sys 2 734003200
The D means that as soon as the last process attached to the shared memory segment dies, the shared memory segment will be cleaned up. In this case, the OS is able to manage the removal of the segment because has been notified that it should remove segment when the number of attached processes, NATTCH, decrements to zero.
Workarounds?
So, now that you know a bit about how shared memory is allocated, and some of the reasons that your programs might not be able to allocate as much as is needed, is there anything you can do about this? Here are a couple of things to keep in mind when dealing with programs that use shared memory:
1) If your program is running out of shared memory, and you determine that fragmentation (i.e. enough memory, just not in one chunk) is the problem, then try starting the program as soon after system boot time as possible. This allows your program to get its memory before the shared memory pool is too fragmented.
2) If starting the program soon after system boot time doesn't solve the problem, see if your application has the ability to create more smaller sized shared memory segments, as opposed to fewer larger sized segments. This might effect application performance, but might make the difference between the application running or not.
3) If you are writing your own programs, make sure you write good error handling routines so that the shared memory you allocate can be returned to the shared memory pool in the event of unexpected program termination.
4) Create a hard copy printout from the ipcs(1) command when your system is functioning normally. This will allow you to make comparisons when you think that you might have extraneous segments lying around. It's possible that the programs running on your system will use the same shared memory key each time, so it should be easy to spot shared memory segments which aren't in use.
The Future
Hopefully, this article has helped to improve the reader's understanding of shared memory, and provide some ways that the system administrator can work with shared memory within the limits of a 32-bit operating system. Granted, the problems described in this paper will be problems of the past with the introduction of the 64-bit version of Hewlett-Packard's next version of HP-UX, 11.0. However, this author is sure that the shared memory characteristics described above will be seen for many years to come, on the 32-bit versions of HP-UX. Who's to say the industry won't soon find 264 memory addresses inadequate for the systems of the future? After all, didn't 64 KB of RAM used to be enough?