Ever want to wreck a $500,000 piece of equipment and not get into trouble? Who doesn't!? I recently got some additional training at work on the Checkpoint 41000 and 61000 series appliances. These are some pretty big and beefy, data center grade appliances. Well, after the training completed, we had a chance to fool around with the device, and here's what I discovered...
Before we really dig in to what I found, we need to clarify a few details about the 41k/61k appliances and how they operate. This line of appliances is meant for LARGE amounts of traffic and throughput, where the chassis itself offers built-in redundancy and load balancing across multiple physical blades. Each physical "blade" or module (SGM to be specific) is technically it's own computer, with 40 CPU cores, and a minimum of 24GB RAM. A fully-populated 61k chassis can accomodate 12 physical blades (the 41k is only 4), meaning that a fully-populated 61k can have a total of 480 cores (@ 2.4GHz each), and 1024GB (1TB) RAM. Due to the large amounts of RAM involved, the Gaia Linux kernel *NEEDS* to run a 64-bit kernel (as a 32-bit kernel is limited only 4GB or RAM).
So how will we make it run poorly? We will simply limit the amout of RAM that can be used by making the 61k boot into a 32-bit kernel. In this manner, the device will still boot, and even pass traffic! But because the 32-bit kernel will be limited to a mere 4GB of RAM, the performance will be absolute garbage in comparison. Let's take a look at how to do so and some results of this.
Before making any changes, let us first take a look at some basic RAM usage. Keep in mind, that this device is in a lab, and the traffic load consists of 1 host streaming some YouTube videos.
[Expert@Lab61k-ch02-03]# free -m
total used free shared buffers cached
Mem: 64280 7720 56559 0 55 280
-/+ buffers/cache: 7383 56896
Swap: 67593 0 67593
We can see here that the device is using 7GB of RAM just to get things going. So now, let us set the boot kernel to 32-bit.
We need to do this from within clish, and fortunately, the commands can be auto-completed by pressing the 'Tab' key. According to Checkpoint's documentation, this is merely 1 single command: "set edition 32-bit". Let's try:
gLab61k-ch02-03 > set ed
gLab61k-ch02-03 > set ed
CLINFR0329 Invalid command:'set ed '.
gLab61k-ch02-03 > set ed
gLab61k-ch02-03 > set edition
32-bit 64-bit
gLab61k-ch02-03 > set edition
gLab61k-ch02-03 > set edition 32-bit
CLINFR0699 Invalid command.
gLab61k-ch02-03 >
Oooh, it looks like Checkpoint already thought of this... First off, the 'set ed<TAB>' is not auto-completing (as we would expect). Which is funny because the 'set edition<TAB>' IS auto-completing, and giving us the options to complete the command. But the complete command isn't working... It would seem that Checkpoint has covered this avenue.
Fortunately for us (or unfortunately for 61k admins), there is another way:
- First, we need to edit a file: /etc/appliance_config.xml:
[Expert@Lab61k-ch02-03]# vi /etc/appliance_config.xml
- Look for the section: <edition>
<edition><default>64bit</default><default_upgrade>64bit</default_upgrade><install_32bit><min_mem>0</min_mem></install_32bit><install_64bit><min_mem>4097</min_mem></install_64bit></edition>
- Change the <default> and <default_upgrade> from "64bit" to "32bit"
- Now, since the SGMs are all technically seperate devices, we need to copy the file to all SGMs, which can easily be done via command:
asg_cp2blades /etc/appliace_config.xml /etc/appliace_config.xml
- Reboot the SGMs. You can either reboot them all individually (to allow for failovers and continued traffic), or the whole device (and who cares about connectivity).
- Once rebooted, you can now run the 'set edition' CLISH command:
set edition 32-bit
- Once again, reboot the device.
- And Voila, it's running 32-bit instead of 64.
At this point, the device will be VERY susceptible to any kind of DoS attack. This is because the device will only be able to access a (in comparison) very small amnount of RAM, thus greatly limiting the connections table. Once this happenes, there won't be any immediately obvious cause for the connectivity issues. Though a thorough examination of the device would/should show that the device is only running 32-bit, so it SHOULD be eventually found.
Notes and Caveats
- While this was tested and performed in a lab setting, I was not able to reproduce it again (due to limited access) to further document it's effect
- This could be due to a number of factors I am still investigating
- Also due to the limited access, I am not able to truly test/verify what would happen when we filled the connection table
- Would it behave like a regular gateway whose connection table has maxxed out?
- Would it crash because of the memory limitations?
- I referenced a few Chekpoint SKs:
- The command '
uname -a' did indeed show that it was running a 32-bit kernel, but I was not able to truly test the memory limits- What, exactly, would happen once that 32-bit 4GB RAM limit was reached/breached?
In order to really further proceed and further document this, I would need some long-ish term access to a 61k, where I can test different versions of the Checkpoint software, different patch levels, and such... It is possible that what I had found is a (possibly undocumented) bug found only specific versions. If you have a 41k/61k aapliance you are willing and able to break and you want to add more info to this, let me know what you find in the comments section below, and I'll amend the article as needed.
