Day to day, I deal with enterprise grade network equipment. Products made by companies like Cisco, HPE Aruba, and FortiNet. Equipment that is at home in a small factory's network closet and an enterprise data center.
I can't deny that it is a lot of fun working with this equipment, but there is always something missing.
I mean, they do everything they are supposed to do. Powerful pieces of equipment that measure throughput in millions of packets per second. Super computers compared to what I grew up around. The sorts of things that I dreamed about as a kid.
But there is something alluring about the little guys. The weird, inexpensive computers that do neat things. You know, the Raspberry Pis and their clones like the Banana Pis and Orange Pis. The ODROIDs by HardKernel. The LattePandas and Arduinos. The weird SOCs that can't possibly see much usage in the real world.
I've always had a soft spot in my heart for this stuff.
I recently came across a ZyXEL GS1900-24HPv2, which is a relatively inexpensive 24 port managed switch with PoE. This is interesting to me because if it is managed, it must have enough power to run some sort of operating system. Obviously, this made me wonder what makes it tick.
Due to my location, it would be a week or so before it was in my hands. Why not take a look at the firmware and see what we can figure out.
The firmware for this switch was pretty easy to find. A quick search on their site even showed that the newest version was only about 4 months old! That was unexpected! I just had a feeling that it most likely hadn't ever been updated.
I had a feeling that this firmware file was nothing more than an archive that gets written directly to the internal flash on the switch. Not exactly the same, but pretty similar to the image that you would write to a flash drive to install your favorite Linux distro. There is really only one way to find out, so let's get at it.
The firmware download comes in the form of GS1900-24HPv2_2.80(ABTP.0)C0.zip. This archive comes with a few interesting files.
unzip GS1900-24HPv2_2.80(ABTP.0)C0.zip
Archive: GS1900-24HPv2_2.80(ABTP.0)C0.zip
creating: GS1900-24HPv2_Firmware_V2.80(ABTP.0)C0/
inflating: GS1900-24HPv2_Firmware_V2.80(ABTP.0)C0/GS1900-24HPv2_ReleaseNote_V2.80(ABTP.0)C0.pdf
inflating: GS1900-24HPv2_Firmware_V2.80(ABTP.0)C0/GS190024HPv2_V2.80(ABTP.0)C0-foss.pdf
inflating: GS1900-24HPv2_Firmware_V2.80(ABTP.0)C0/runtime-GS1900-24HPv2-V2.80(ABTP.0).bixThis is license information and release notes for this firmware. The license is not remarkable, but there is this gem in the release notes:
If user tries to restore an illegal configuration (unofficial), login will always show authentication fail.
I'm not entirely sure what this means, but my instinct is saying that this switch will soft brick itself if you try to do anything that is "unsupported."
The really interesting file is runtime-GS1900-24HPv2-V2.80(ABTP.0).bix. I've not heard of a bix file, but let's poke at it.
The easiest first step here is to simply walk through the binary file with binwalk. Binwalk is a super cool tool that searches for magic headers within another file to attempt to locate embedded data. It is also able to extract the data that it finds for further analysis.
Binwalk is a fast, easy to use tool for analyzing, reverse engineering, and extracting firmware images.
binwalk -B runtime-GS1900-24HPv2-V2.80\(ABTP.0\).bix
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 bix header, header size: 64 bytes, header CRC: 0x500CFA4F, created: 2023-10-16 10:38:39, image size: 6084981 bytes, Data Address: 0x80000000, Entry Point: 0x8026F000, data CRC: 0x1600B50E, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: gzip, image name: "V2.80.0"
64 0x40 gzip compressed data, maximum compression, has original file name: "vmlinux_org.bin", from Unix, last modified: 2023-10-16 02:38:39
1172071 0x11E267 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 7988 bytes
1442900 0x160454 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 131072 bytes
1748357 0x1AAD85 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 1772 bytes
1748910 0x1AAFAE LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 3180 bytes
1832939 0x1BF7EB LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 30176 bytes
1903834 0x1D0CDA LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 131072 bytes
2105782 0x2021B6 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 14656 bytes
2195407 0x217FCF LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 33152 bytes
2262005 0x2283F5 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 64456 bytes
2761485 0x2A230D LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 28108 bytes
3231753 0x315009 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 35133 bytes
3290447 0x32354F Zlib compressed data, best compression
3343697 0x330551 Zlib compressed data, best compression
3396467 0x33D373 Zlib compressed data, best compression
3449347 0x34A203 Zlib compressed data, best compression
3502160 0x357050 Zlib compressed data, best compression
3836698 0x3A8B1A Zlib compressed data, best compression
3844062 0x3AA7DE Zlib compressed data, best compression
4032946 0x3D89B2 Zlib compressed data, best compression
4400040 0x4323A8 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 17880 bytes
5188333 0x4F2AED LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 36496 bytes
5449690 0x5327DA LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 45912 bytes
5729393 0x576C71 LZMA compressed data, properties: 0x5D, dictionary size: 131072 bytes, uncompressed size: 14440 bytesThere is sort of a lot going on here. Some of the highlights:
I have never heard of a bix header, or even a bix file. It must be a thing if binwalk was able to pick this up. Some quick searching indicated that this file type was an experimental compression scheme invented by Igor Pavlov. It was apparently a predecessor of 7z and never had wide usage.
I spent quite some time trying to figure out how to decompress a bix archive. There were some ancient tools that were originally hosted on Angelfire but no other documentation to be found. This will become a theme of this entire series.
While that was an exercise in frustration, binwalk was able to identify the file, so maybe it can extract it as well.
Using binwalk to extract files from an archive is pretty straightforward. We just throw the -e switch for extract, and -M for matryoshka.
binwalk -e -M runtime-GS1900-24HPv2-V2.80(ABTP.0).bixThis extracts quite a bit, and honestly, most of it doesn't make sense yet. There are a few embedded png files that appear to show the front of a Zyxel switch, although not this model. There are also quite a few binary files with super helpful filenames.
ls -Al
-rw-r--r-- 1 vivek vivek 7988 Mar 7 18:42 11E267
-rw-r--r-- 1 vivek vivek 4912974 Mar 7 18:42 11E267.7z
-rw-r--r-- 1 vivek vivek 131072 Mar 7 18:42 160454
-rw-r--r-- 1 vivek vivek 4642145 Mar 7 18:42 160454.7z
-rw-r--r-- 1 vivek vivek 4952 Mar 7 18:42 1AAD85
-rw-r--r-- 1 vivek vivek 4336688 Mar 7 18:42 1AAD85.7z
-rw-r--r-- 1 vivek vivek 3180 Mar 7 18:42 1AAFAEI am not going to pretend to be an expert at using binwalk, so all of the intricacies are out of scope for this article. I would like to point out that while these filenames might appear random, they really aren't.
I'll use 11E267 as my example. This is the hexadecimal notation of how many bytes into the original file that this file was located. A quick conversion shows that this file was found 1172071 bytes into the original. Maybe not super helpful in the scheme of things, but definitely better than being random.
The extraction also resulted in a couple of nested directories being created:
A total of 47 binary files were extracted by binwalk. I could spend time examining each of them, but one really piqued my interest. _vmlinux_org.bin.extracted.
Drilling down into this directory reveals the most interesting bit. The initial root filesystem, the cpio-root.
When the Linux kernel first starts it does not have all of the drivers that it might need to boot the system. For example, it may not be able to mount a network filesystem, or an encrypted volume if that volume needs a password.
For most distributions, kernel modules are the biggest reason to have an initramfs. In a general distribution, there are many unknowns such as file system types and disk layouts.
To get around this, and to know how to proceed booting, Linux embeds an initramfs into the kernel. This is a cpio file archive that contains what can essentially be a complete distribution. After mounting this archive as the root filesystem, it then checks for the presence of a file called init and attempts to run it as PID 1. In most modern, every day Linux distributions, this would be systemd.
Sure enough, the extracted cpio files do indeed look like a Linux filesystem, and there is even an init file.
ls
bin etc init mnt modsqfs proc sqfs sys usr webtmp
dev home lib mntlog modsqfs.img sbin sqfs.img tmp var
ls -Al | grep init
lrwxrwxrwx 1 vivek vivek 9 Mar 7 18:42 init -> sbin/initWe can see here that init is linked to sbin/init. In turn, sbin is actually linked to bin. When we take a look there, we can see that init is actually a link to busybox.
ls bin -Al | grep init
lrwxrwxrwx 1 vivek vivek 14 Mar 7 18:42 init -> ../bin/busybox
lrwxrwxrwx 1 vivek vivek 9 Mar 7 18:42 initd -> /dev/null
Busybox uses a script called /etc/inittab for initialization. There is not much here other than the next clue.
::sysinit:/etc/rc
ttyS0::respawn:/bin/sh/etc/rc is quite a bit more lengthy so I won't post the whole thing. It is cleanly written though so pretty easy to follow. The general flow of what happens:
I would also like to highlight a gem that I found. When this switch was ordered, there was an option for a bit more to have Zyxel One Network included. I opted out of this, but it looks like the file still exists, sort of.
#if test -f "/bin/zon"; then
# zon &
#fils /bin -Al | grep zon
lrwxrwxrwx 1 vivek vivek 9 Mar 7 18:42 zon -> /dev/null
Poking around this filesystem some more, it is clear that it is not the full system. There are quite a few binaries, along with configuration files. However, there are also a lot of empty directories, and a lot of things that are symlinked to /dev/null.
At this point, I can't guarantee anything, but I have a feeling that the missing files are contained in sqfs.img and modsqfs.img. My gut is saying that these are SquashFS images and are mounted with the mountfs command that is called during the boot process.
Unfortunately, mountfs is a binary file and not a script. It would have been really cool to read a script, but let's see what we can learn from it.
Using dhex, it is immediately obvious that this is an ELF (Executable and Linking Format), or a Linux executable. It was compiled with uClibc, and it has a surprising amount of human-readable text in it.
Let's see if it's easy to pull out the English text.
strings mountfs
/lib/ld-uClibc.so.0
_init
_fini
__uClibc_main
memcpy
open
ioctl
close
malloc
read
free
strncpy
strncmp
bootmsg
atoi
memset
system
sprintf
_DYNAMIC_LINKING
__RLD_MAP
libgcc_s.so.1
_gp_disp
libc.so.0
_DYNAMIC
_GLOBAL_OFFSET_TABLE_
_ftext
_fdata
_edata
__bss_start
_fbss
_end
@ !$
@ !$
@ !$
@ !$
@ !$
@ !$
bootmsg
/dev/mtdchar4
mount -t devpts devpts /dev/pts > /dev/null
Failed
Mount DEV File System....%s
mount -t proc proc /proc > /dev/null
Mount PROC File System........%s
mount -o loop /sqfs.img /sqfs > /dev/null
Mount Main SQFS File System........%s
mount -o loop /modsqfs.img /modsqfs > /dev/null
Mount Module SQFS File System....%s
mount -t jffs2 -o rw,sync %s %s > /dev/null
/dev/mtdblock3
/mnt
Mount CFG JFFS2 File System....%s
/dev/mtdblock4
/mntlog
Mount LOG JFFS2 File System....%s
.shstrtab
.interp
.reginfo
.dynamic
.hash
.dynsym
.dynstr
.text
.MIPS.stubs
.rodata
.data
.rld_map
.got
.sbss
.bss
.pdr
.mdebug.abi32Holy crap! I haven't examined the machine code or anything, but it really looks like there is a list of devices to be mounted using the regular old mount command.
mount -t devpts devpts /dev/pts > /dev/null
Failed
Mount DEV File System....%s
mount -t proc proc /proc > /dev/null
Mount PROC File System........%s
mount -o loop /sqfs.img /sqfs > /dev/null
Mount Main SQFS File System........%s
mount -o loop /modsqfs.img /modsqfs > /dev/null
Mount Module SQFS File System....%s
mount -t jffs2 -o rw,sync %s %s > /dev/null
/dev/mtdblock3
/mnt
Mount CFG JFFS2 File System....%s
/dev/mtdblock4
/mntlog
Mount LOG JFFS2 File System....%sIf this is doing what I think it's doing, the system mounts /dev/pts, /proc, then /sqfs.img and /modsqfs.img. It then mounts a couple JFFS2 filesystems that are stored on the flash storage. (JFFS2 is pretty much only used for flash chips. It is what it was designed for.)
At this point there is no easy way for me to dump the contents of the JFFS2 filesystems, but eventually... in the next article.
We need to find out what is contained in the sqfs.img file that is mounted to /sqfs. I don't actually see how the files on this mount are accessed in the init script, but it may give us clues just looking at it.
binwalk -B sqfs.img
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 Squashfs filesystem, big endian, lzma signature, version 3.1, size: 3895415 bytes, 406 inodes, blocksize: 131072 bytes, created: 2023-10-16 02:38:19Sweet, it is a SquashFS image. SquashFS is a read-only filesystem that is used all over the place. You don't generally come across SquashFS images in day to day activities, but, well this quote will say it better than I can:
Squashfs is used by the Live CD versions of Arch Linux, Debian, Fedora, Gentoo Linux, Linux Mint, openSUSE, Salix, Ubuntu, NixOS, Clonezilla, Kali Linux, KDE neon and on embedded distributions such as the OpenWrt and DD-WRT router firmware. It is also used in Chromecast, in Tiny Core Linux for packaging extensions, and for the system partitions of some Android releases (Android Nougat). The AppImage project, which aims to create portable Linux applications, uses Squashfs for creating AppImages. The Snappy package manager also uses Squashfs for its ".snap file format".
Since SquashFS is such a common image format, it can be mounted or extracted pretty easily.
I'm going to copy this file to its own directory so that I don't pollute the extracted cpio-root directory. After that, I'll create a mount point and see what I can do.
mkdir mnt
sudo mount --type="squashfs" --options="loop" --source="sqfs.img" --target="mnt"
mount: mnt: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.Wait, what?
Maybe I can just unsquash it?
unsquashfs sqfs.img
FATAL ERROR: Can't find a valid SQUASHFS superblock on sqfs.imgOk... After several more attempts and a bunch of reading, I found out that some vendors alter the SquashFS magic signature. I'm not sure why they do this, but it is a thing, and there is a tool to combat this. Sasquatch.
The sasquatch project is a set of patches to the standard unsquashfs utility (part of squashfs-tools) that attempts to add support for as many hacked-up vendor-specific SquashFS implementations as possible.
sasquatch sqfs.img
SquashFS version [768.256] / inode count [-1778319360] suggests a SquashFS image of a different endianess
Non-standard SquashFS Magic: qshs
Reading a different endian SQUASHFS filesystem on sqfs.img
Parallel unsquashfs: Using 1 processor
Trying to decompress using default gzip decompressor...
Trying to decompress with lzma...
Detected lzma compression
385 inodes (466 blocks) to write
[=======================================================================================================|] 466/466 100%
created 383 files
created 21 directories
created 2 symlinks
created 0 devices
created 0 fifos
This worked, and it extracted into another partial root filesystem. This time, we have a few new options available:
Some of the binaries that were linked to /dev/null appear to be present here. It is also pretty noteworthy that /home/web appears to be the root of the web interface for the switch!
This still doesn't explain everything. At this point, I know that / has quite a few binaries that are symlinked to /dev/null. I also know that /home/web is symlinked to /dev/null. sqfs.img is mounted to /sqfs, and it has the data that needs to go in /home/web, but how is that actually done?
The only initialization steps that haven't been investigated are found in the rc script that we found earlier.
#
# start application
#
initd
inetd &
polld &
timed &
if test -f "/bin/macr"; then
macr &
fi
if test -f "/bin/ksid"; then
ksid &
fiI'm thinking that initd is the natural choice here, but I have no idea what macr and ksid are. So we may have some more digging to do.
Since initd is run first and it has "init" in the name, I decided to check out that file first.
strings bin/initd | lessThere is way too much output to show here. There are a few things that stand out though.
mount -t tmpfs -o size=8m tmpfs /tmp/upload
pwdrecov
admin
1234
zon &
/mnt/startup-configIt looks like a temporary filesystem is created for uploads. This is limited to 8 Mb, so any uploaded firmware will need to be less than that in size. There is also "pwdrecov" followed by the default username and password.
It also looks like a reference to either a script or program at /mnt/startup-config. I had discovered earlier that the CFG JFFS2 filesystem is mounted to /mnt.
mount -t jffs2 -o rw,sync %s %s > /dev/null
/dev/mtdblock3
/mnt
Mount CFG JFFS2 File System....%sWe are going to need to get access to /dev/mtdblock3. This will be fun.
