While analyzing the boot process of a Zyxel GS1900-24HPv2 Managed Switch, it became apparent that I would need to analyze the flash storage on the board. Lacking access to the operating system, and with no way to connect a display or keyboard, I knew this was going to be a fun project.
In the previous article, I started Exploring the Firmware of a Zyxel Switch. After a bit of poking around it became apparent that I would need to take a look at the contents of the flash chips to really understand how the system worked. The extracted firmware has most of a complete Linux system, but it really doesn't look like it contains everything.
My initial analysis showed that data on a flash chip, referenced as /dev/mtdblock3, is mounted to /mnt. A bit later in the start up process, it looks like a script mounted to /mnt/startup-config is run. This has to be the missing link that sets up the remaining filesystems to create a usable system.
How can I get to the data stored on this memory technology device?
There are a few different ways that we can get to the data on /dev/mtdblock3. The first few options that come to mind:
Each of these options have some additional considerations, and some aren't going to be possible yet.
This would be the easiest method by far. Just login to the system like any other and take a look around. Unfortunately, when logging in to this switch, the shell is very Cisco-like, and there is no clear way to break out of that. This is not a standard shell, but one created just for the purpose of managing a network device. This option will not work out of the box.
This is the coolest option. Why not build out our own tool set just to further reverse engineer this switch? That is some serious street cred.
This also takes the most understanding of the chipset. It is relatively easy to compile a kernel, but not quite as easy to compile a kernel for a different architecture, along with the appropriate modules. I am going to have to skip this option for now, but will almost certainly come back to it.
This is another solid option. Work has even already been done to get OpenWrt ported to this system. This would likely be an easy solution, but it also feels like cheating at this point. This series is about figuring out what we can do on our own, and with this option, someone has already done the hard work.
Das U-Boot, the bootloader on this switch, has the functionality built in to read memory addresses. It is extremely low level. It feels right. This is what I want to do first.
Since we aren't at the point yet where we can boot our own system on here, we will need to work with the current tools. Luckily, U-Boot has the ability to display the contents of memory.
Das U-Boot (subtitled "the Universal Boot Loader" and often shortened to U-Boot; see History for more about the name) is an open-source boot loader used in embedded devices to perform various low-level hardware initialization tasks and boot the device's operating system kernel.
Just like Wikipedia says, U-Boot is a boot loader for embedded devices. It serves a similar purpose as GRUB serves for desktop Linux. When an embedded device is first powered on it will start execution of U-Boot, which in turn is configured to start the kernel with appropriate settings for the board it is on.
Since U-Boot is designed to run at such a low level, and it is used on embedded devices with various architectures, it is frequently compiled with a decent set of features for interacting with the hardware directly. The exact process will differ from board to board, but for the Zyxel GS1900-24HPv2, we just need to hook up to the serial port and press space to interrupt the boot process.
There are quite a few different tools that you could use for serial communication. I've recently started using minicom after a suggestion from CubicleNate. Any instructions in this article are assuming that minicom is being used, so your mileage might vary with other programs. For a quick run down of using minicom you can check out CubicleNate's blathering, Configuring a Cisco switch from a Linux Terminal with Minicom.
The process here is pretty simple. I need a serial connection to the switch while it is booting. Just after power up of the switch, the switch will prompt to press space to abort booting. Doing this will give me access to the serial console for U-Boot.
To ensure that I have the connection ready to go in time, I'll follow this process:
minicom -b 115200 -D /dev/ttyUSB0
Welcome to minicom 2.8
OPTIONS: I18n
Port /dev/ttyUSB0
Press CTRL-A Z for help on special keys
U-Boot Version: 2.0.2.5 (Aug 31 2021 - 02:24:56)
CPU: 500MHz
DRAM: 128 MB
FLASH: 16 MB
Model: ZyXEL_GS1900_24HPv2
SN: S222L31002671
MAC: D8:EC:E5:BB:5E:08 - D8:EC:E5:BB:5E:22
Press SPACE to abort boot script: 0
RTL838x#Sweet, just like that, we have a U-Boot shell!
U-Boot's shell has a bunch of built in commands. These differ from installation to installation, but most share some common ones. I've truncated the list a bit, but let's see what we can do.
RTL838x# ?
? - alias for 'help'
bdinfo - print Board Info structure
boardid - boardid - Get/Set board model id
boot - boot default, i.e., run 'bootcmd'
boota - boota - boot application image from one of dual images partition automatically
bootd - boot default, i.e., run 'bootcmd'
bootm - boot application image from memory
bootp - boot image via network using BOOTP/TFTP protocol
cp - memory copy
crc32 - checksum calculation
cst - cst - customer commands
echo - echo args to console
editenv - edit environment variable
env - environment handling commands
erase - erase FLASH memory
flerase - Erase flash partition
flinfo - print FLASH memory information
flshow - Show flash partition layout
go - start application at address 'addr'
help - print command description/usage
iminfo - print header information for application image
imxtract- extract a part of a multi-image
loadb - load binary file over serial line (kermit mode)
loady - load binary file over serial line (ymodem mode)
loop - infinite loop on address range
md - memory display
mm - memory modify (auto-incrementing address)
mw - memory write (fill)
nfs - boot image via network using NFS protocol
nm - memory modify (constant address)
ping - send ICMP ECHO_REQUEST to network host
printenv- print environment variables
printsys- printsys - print system information variables
reset - Perform RESET of the CPU
rtk - rtk - Realtek commands
saveenv - save environment variables to persistent storage
savesys - savesys - save system information variables to persistent storage
setenv - set environment variables
setsys - setsys - set system information variables
sf - SPI flash sub-system
source - run script from memory
tftpboot- boot image via network using TFTP protocol
version - print monitor, compiler and linker version There are a ton of things we can do, even when looking at this shortened list. I am going to focus on the following:
Honestly, I would be lying if I tried to say that I haven't already used all of these. All of this was tested out before writing this article at all :) I've run most of the commands available just to see what can be done.
While doing everything possible would be a really great time, I need to reel myself in and focus on getting to that juicy flash filesystem.
RTL838x# version
U-Boot Version: 2.0.2.5 (Aug 31 2021 - 02:24:56)
mips-linux-uclibc-gcc (GCC) 3.4.4 mipssde-6.03.00-20051020
GNU ld version 2.15.94 mipssde-6.03.00-20051020This is neat. uClibc was used to compile U-Boot, and it appears to have created with the MIPS Software Development Environment.
RTL838x# printenv
HTPLog=0
baudrate=115200
boardmodel=ZyXEL_GS1900_24HPv2
bootargs=console=ttyS0,115200 mem=64M quiet
bootcmd=boota
bootdelay=1
ethact=rtl8380#0
ethaddr=D8:EC:E5:BB:5E:08
ipaddr=192.168.1.1
netmask=255.255.255.0
runHTP=0
serverip=192.168.1.111
stderr=serial
stdin=serial
stdout=serial
Environment size: 314/1020 bytesAlright, this is neat to see. We've got the kernel boot arguments, default U-Boot boot command, and even the 1 second in which you have to press space to abort booting.
RTL838x# bdinfo
boot_params = 0x83DDFB10
memstart = 0x80000000
memsize = 0x08000000
flashstart = 0xB4000000
flashsize = 0xA5A5A5A5
flashoffset = 0x00000000
ethaddr = D8:EC:E5:BB:5E:08
ip_addr = 192.168.1.1
baudrate = 0 bpsNice. Addresses for accessing RAM and flash filesystems. Someday I should write an article that goes over how to grok this information and link to it here.
Without that article, if you don't intuitively understand what you are looking at, this is what I get out of this:
RTL838x# flshow
=============== FLASH Partition Layout ===============
Index Name Size Address
------------------------------------------------------
0 LOADER 0x40000 0xb4000000-0xb403ffff
1 BDINFO 0x10000 0xb4040000-0xb404ffff
2 SYSINFO 0x10000 0xb4050000-0xb405ffff
3 JFFS2_CFG 0x100000 0xb4060000-0xb415ffff
4 JFFS2_LOG 0x100000 0xb4160000-0xb425ffff
5 RUNTIME1 0x6d0000 0xb4260000-0xb492ffff
6 RUNTIME2 0x6d0000 0xb4930000-0xb4ffffff
======================================================Sweet! This shows the exact address that the infamous JFFS2_CFG is located! That is what I'm looking for.
I understand that this is something that can make your eyes glaze over. For now, we can think of the switch as being a book. There are 7 chapters in this book, and flshow is telling us where to flip the book open to for each chapter.
md will show the contents of whatever address it is given. The address is the hexadecimal numbers that are shown in the various commands above.
This will be explored more in the next section.
The other commands are all important and useful. I have a suspicion that they are going to be used later, but right now, not as much.
Alright, I'm starting to get excited. We have a table of addresses for data, and we have the md command to show us that data.
Let's take a closer look at how this works. The flshow command shows that the BDINFO partition is at 0xb4040000. We just put that into md.
RTL838x# md 0xb4040000
b4040000: b85f12dd 4854504c 6f673d30 00626175 ._..HTPLog=0.bau
b4040010: 64726174 653d3131 35323030 00626f61 drate=115200.boa
b4040020: 72646d6f 64656c3d 5a795845 4c5f4753 rdmodel=ZyXEL_GS
b4040030: 31393030 5f323448 50763200 626f6f74 1900_24HPv2.boot
b4040040: 61726773 3d636f6e 736f6c65 3d747479 args=console=tty
b4040050: 53302c31 31353230 30206d65 6d3d3634 S0,115200 mem=64
b4040060: 4d207175 69657400 626f6f74 636d643d M quiet.bootcmd=
b4040070: 626f6f74 6100626f 6f746465 6c61793d boota.bootdelay=
b4040080: 31006574 68616374 3d72746c 38333830 1.ethact=rtl8380
b4040090: 23300065 74686164 64723d44 383a4543 #0.ethaddr=D8:EC
b40400a0: 3a45353a 42423a35 453a3038 00697061 :E5:BB:5E:08.ipa
b40400b0: 6464723d 3139322e 3136382e 312e3100 ddr=192.168.1.1.
b40400c0: 6e65746d 61736b3d 3235352e 3235352e netmask=255.255.
b40400d0: 3235352e 30007275 6e485450 3d300073 255.0.runHTP=0.s
b40400e0: 65727665 7269703d 3139322e 3136382e erverip=192.168.
b40400f0: 312e3131 31007374 64657272 3d736572 1.111.stderr=serYeah! That is literally the same data as printenv! Let's see the content of JFFS2_CFG.
RTL838x# md 0xb4060000
b4060000: 19852003 0000000c f060dc98 1985c001 .. ......`......
b4060010: 0000002b 3e422427 00000001 00000000 ...+>B$'........
b4060020: 00000002 61dd50d2 03040000 9248306f ....a.P......H0o
b4060030: 707eb1d7 6c6f67ff 1985c002 00000044 p~..log........D
b4060040: a4ef223e 00000002 00000001 000041ed ..">..........A.
b4060050: 03f603f7 00000000 61dd50d2 61dd50d2 ........a.P.a.P.
b4060060: 61dd50d2 00000000 00000000 00000000 a.P.............
b4060070: 00000000 00000000 81c96bae 1985c002 ..........k.....
b4060080: 00000044 a4ef223e 00000003 00000001 ...D..">........
b4060090: 000041ed 00000000 00000000 61cf9980 ..A.........a...
b40600a0: 61cf9980 61cf9980 00000000 00000000 a...a...........
b40600b0: 00000000 00000000 00000000 04256569 .............%ei
b40600c0: 1985c001 0000002b 3e422427 00000001 .......+>B$'....
b40600d0: 00000001 00000003 61cf9980 03040000 ........a.......
b40600e0: dcd0a399 11cc1556 737368ff 1985c002 .......Vssh.....
b40600f0: 00000044 a4ef223e 00000003 00000002 ...D..">........Hmm...
Well, that is not so obvious what is going on. I mean, I can see both .log and ssh in that output, but the remainder is just some binary. That's not so useful.
Here is what I've gotten out of this so far:
If we do some quick math we can discover the size of this flash partition.
(End Address) - (Start Address) = (Size)
0xb415ffff - 0xb4060000 = FFFFF
FFFFF = 1,048,575This shows that this flash partition, JFFS2_CFG, is 1 MB. md will only show us 256 bytes at a time. So all we need to do is run md 4096 times and we'll be able to capture all of it.
There is also another obstacle to overcome. How can these hex values be converted to the binary they represent?
I'm not sure about you, but I'm not going to manually run this command 4096 times, copy and paste the output, then try to manually convert it to binary. I'm going to do it the fun way.
As far as this article is concerned, this is it. We are at the moment that things come together. We know what needs to be done... but how?
Expect. Oh, and Python.
Expect is used to automate control of interactive applications such as Telnet, FTP, passwd, fsck, rlogin, tip, SSH, and others. Expect uses pseudo terminals (Unix) or emulates a console (Windows), starts the target program, and then communicates with it, just as a human would, via the terminal or console interface.
I've written a pair of scripts for this, which can be found in my zyxel-flash-dump repo on GitHub.
The first script, minicom.exp, is an Expect script that launches minicom. This script watches for the U-Boot prompt of RTL838x#. Each time it sees the prompt, it will increase the memory address by 256 bytes (or 0x100), then run the md command again with the new address. All output is saved to a file.
The second script, jffs2-convert.py, takes the output file from minicom.exp, parses it, and converts the hexadecimal values to binary. The binary data is then written to jffs2_cfg.img, giving us the sought after JFFS2_CFG flash partition.
For the unlikely event that GitHub goes down and this article is up, here is the expect script that I used.
#!/usr/bin/expect -f
#
# Originally written to dump the contents of flash on a Zyxel GS1900-24HPv2.
# This could pretty easily be adapted to dump other ranges of memory.
# Set variables
set address 0xb4060000
set end_address 0xb415ffff
set jffs_raw "jffs2_raw.dump"
set usbtty "/dev/ttyUSB0"
# Open minicom
spawn bash -c "time minicom -b 115200 -D $usbtty | tee $jffs_raw"
# Wait for minicom to start
sleep 1
# Send command to enter minicom's command mode
send "\r"
expect "RTL838x#"
sleep 1
# Loop until end address is reached
while {$address <= $end_address} {
# Send memory dump command. Make sure that the address is formatted
# as a hex number. This switch will crash if a decimal number is used.
send "md [format %x $address]\r"
# Wait for command output
expect "RTL838x#"
# Increment address
set address [expr {$address + 0x100}]
}
# Wait for minicom to exit
expect eofFor converting the output of the above Expect script to an image file, this Python script was used.
Note: This script was edited after this article was originally published. I found that the original script was not parsing the hex data correctly and building corrupted images.
#!/usr/bin/python
import re
output_file = 'jffs2_cfg2.img'
binary_data = b''
hex_pattern = re.compile(r'[0-9a-fA-F]{8}')
with open("jffs2_raw.dump", "r") as file:
for line in file:
# Check if the line contains "RTL838x# md", if so, skip
if "RTL838x# md" in line:
continue
# Find the index of ":" to locate the end of the memory address
address_end = line.find(":")
if address_end == -1:
continue
# Skip any lines that are too short to contain data
if len(line) < 45:
continue
# Extract the part of the line after the memory address
hex_data = line[address_end+1:]
# Extract hexadecimal numbers from the line
hex_values = re.findall(hex_pattern, hex_data)
# Convert to binary and concatenate
for hex_value in hex_values:
binary_data += bytes.fromhex(hex_value)
with open(output_file, 'wb') as file:
file.write(binary_data)
Now that I have an image of the much sought after JFFS2_CFG flash partition, I can take a look and see what the enigmatic /startup-config script does. I just need to mount the image and open the file.
Right?