Last updated at Fri, 08 Dec 2023 18:25:36 GMT
This post is the fifth in a series, 12 Days of HaXmas, where we take a look at some of more notable advancements and events in the Metasploit Framework over the course of 2014._
Writing portable software is not hard. It's just like walking through a minefield! Getting to the other side, that's the tricky part.
Sure, if you target C, Unix-like systems and GCC or LLVM, you may not run into too many hassles these days. There are still a few annoying differences between BSDs and Linux, but POSIX and better compiler compatibility have been shrinking the gaps. Even the old Unix systems have learned to either adapt or die. Endian and byte-alignment considerations are becoming easier too, with newer ARM, MIPS and PowerPC CPUS are starting to copy the behavior of that the lovable little-endian, byte-access-agnostic x86. Though don't forget to test occasionally on a SPARC machine, just to keep you honest .
But even if you restrict your program to use only POSIX systems and library calls, and your code builds without warning and -Wpedantic on a half dozen compilers, beware the silent killer - error handling. Two system calls may act the same way when everything is ok, but do they both fail the same way? You have to not only make sure that your API calls quack like a duck and walk like a duck, but also blend like a duck. Quiet compilers have a way of creating a false sense of security.
There are plenty of traps. Take asprintf() for instance. It attempts to make formatting string buffers safe and easy, but has 2 ways it can fail, either by returning an integer or setting a pointer. Either (or both) of which are these may be set by different implementations. Or, consider poll, which has a history of not handling errors correctly (Window's WSAPoll failure) or even the checks for broken poll implementations being broken (OS X's [poll() on python]https://github.com/python/cpython/issues/49404).
I recently merged Windows support for LibreSSL, and making things work properly was just as much challenge as making them fail properly. There was an additional challenge as well! The LibreSSL team wanted to add no new #ifdefs to the upstream codebase, lest they make the code harder to audit and test. My eventual goal is to incorporate this work into Meterpreter, updating the aging, yet nicely Heartbleed-free, OpenSSL 0.9.8 that it currently uses.
Easy things first: Random Number Generation
Windows was ahead of its time in this area. When the OpenBSD team added the getentropy() syscall earlier this year in order to provide a failure-free method of seeding arc4random() (the random number generator for LibreSSL), it turned out that acquiring entropy on many popular
OSes was much harder than it needed to be. With Windows, it was extremely simple thanks to CryptGenRandom:
if (CryptAcquireContext(&provider, NULL, NULL, PROV_RSA_FULL,
CRYPT_VERIFYCONTEXT) == 0)
goto fail;
if (CryptGenRandom(provider, len, buf) == 0) {
CryptReleaseContext(provider, 0);
goto fail;
}
CryptReleaseContext(provider, 0);
Some day, when arc4random becomes a POSIX standard and Windows is maybe the odd-OS-out, I'll have a different perspective, but for now, it was a pleasant surprise. But, shortly after, we run into the first porting challenge:
Problem 1: Sockets vs File Descriptors
With POSIX, file descriptors are a common interface for many things, and thus things like sockets, terminals, disks, and files can be manipulated with the same basic system calls like open(), read(), write(), and close(). Windows was designed in a parallel universe and chose the HANDLE as the common abstraction, and if you like handles can use something like OpenFile(), CloseHandle(), etc. There are some alter-ego functions like _open(), _read(), _write() exposed as well that operate on POSIX-like file descriptors, but their scope is far from universal.
Socket APIs in Windows add to the mix if incompatible APIs. While a POSIX programmer might not even blink to write a function that generically takes an 'int' and works on both sockets and files, Windows Sockets look like an 'int' to the compiler, but have their own subset of APIs that only work on sockets, e.g. recv/send/closesocket. Vice-versa, APIs that work on files in POSIX environments don't work on Windows sockets. Take the following contrived example:
void echo4(int accept_fd) {
char buf[4];
int sock = accept(accept_fd, NULL, NULL);
if (sock >= 0 && read(sock, buf, sizeof(buf)) == 4)
write(sock, buf, sizeof(buf));
else
perror("accept or read");
close(sock);
}
This code compiles cleanly on Linux and Mingw-w64 , but it behaves completely differently. In a POSIX system, it will accept a connection on listening socket, read 4 bytes, echo them back and close the connection. On Windows, read() and write() fail and there is a resource leak. The main problem is that accept() on Windows does not return an int, it returns a SOCKET, which the compiler will happily cast to an int anyway, since that's what it is under the covers. But, since read, write and close do not work on SOCKETs, you never know about it (except for the program not working). Or worse, you used recv and send originally, but the close() call is now silently leaks SOCKET objects.
The usual answers to this problem are #ifdefs or using Cygwin, but those are either ugly, or prevent it using LibreSSL as a native Windows port. My solution was to use the C preprocessor to create alternate versions of IO functions that automatically do the right thing for sockets and file descriptors.
static inline ssize_t posix_read(int fd, void *buf, size_t count) {
ssize_t rc = recv(fd, buf, count, 0);
if (rc == SOCKET_ERROR) {
int err = WSAGetLastError();
return err == WSAENOTSOCK ?
_read(fd, buf, count) : wsa_errno(err);
}
return rc;
}
#define read(fd, buf, count) posix_read(fd, buf, count)
This defines a shim posix_read function that exploits a the failure behavior of socket functions. We can then play a guessing game! If the argument was actually a SOCKET (remember, the compiler won't tell you if you used the wrong function), we have guessed the correct call. If it isn't, the error code WSAENOTSOCK tells us to use the read function defined for file descriptors instead.
Problem 2: Error Handling
Have you ever had a program popup a dialog saying "Error: Success" ? POSIX functions set errno on failure, but Windows socket functions instead set an internal variable accessible via WSAGetLastError(). But, errno is still available, so in our example code, using our shim function for 'read' may still report 'Success' for errno, since that is what the 'perror' function looks for. To fix errno's behavior, we define a mapping function that converts WSA* errors into errno-style constants. No more 'ESUCCESS'! :
static int
wsa_errno(int err)
{
switch (err) {
case WSAENOBUFS:
errno = ENOMEM;
break;
...
But we're not out of the woods yet. Windows 'strerror' function does not understand how to format all of the error codes defined in errno.h. So, I overrode that as well:
static inline char *
posix_strerror(int errnum) {
if (errnum == ECONNREFUSED)
return "Connection refused";
return strerror(errnum);
}
#define strerror(errnum) posix_strerror(errnum)
An alternative could have been to remap back to the original WSA error code, and to use FormatMessage to get the localized message. But, these messages turned out to be far longer than the fixed-length error buffers that OpenSSL/LibreSSL allocate for error messages. Rather than have messages like 'The connection that you were about to use has...
Problem 3: select()
The Windows and POSIX APIs for signaling on events are both very capable, but unfortunately much different from each other. The Windows socket API throws portable software a bone by implementing select(), but this is far from POSIX compatible. The catacombs of stackexchange are littered with the bones of programmers surprised that Windows select doesn't work on files, pipes, the console or anything that is not a socket. Surprisingly, even high-level languages like Perl and Python do not bother to hide this difference.
Even if one sticks to sockets, select on Windows has some surprising properties to a POSIX programmer. It is possibly even a little more secure! Among the many design flaws of the select interface in general, it takes a fd_set structure that indicate the sockets one is interested in waiting on. On POSIX systems, this is a actually a bitmap, one bit per descriptor. So, if your descriptor value is 1000, bit 1000 has to be set in the fd_set. This can lead to failures and buffer overflows if you have a file descriptor value larger than the fd_set structure itself, even if only a single descriptor is actually set.
On Windows, an fd_set is instead an array of SOCKETs. That means that the literal value of the SOCKET, when converted to an integer does not matter, though it is limited to about 32 entries before the fd_set is full. For small numbers of sockets, the Windows interface actually helps prevent the overflow! Of course, select is still bad for performance reasons, so we want to fix it.
For readability and security, first we converted all select(2) calls in LibreSSL into poll(2) calls. Poll is a little better than select on POSIX systems, and has the advantage of not having the fd_set overflow issue as well. Windows has a poll-like function as well, but it has the same limitations as select - it only works on sockets. To fix this, I implemented a poll implementation that uses the same mother-may-I SOCKET testing technique to determine if we should do select or WaitForMultipleObjects on passed in values. Here is the code that helps determine what's what:
static int
is_socket(int fd)
{
WSANETWORKEVENTS events;
return (WSAEnumNetworkEvents((SOCKET)fd, NULL, &events) == 0);
}
There are some extra heuristics to determine if a pipe or a console handle have been passed in. In the case of a Windows console, we do not want poll to tell the caller that stdin has data available, when instead it just signaled on a mouse or window event. Because stdin is always blocking, performing a read() without data will hang a program. Since we don't care about these extra events from a console program, poll() automatically discards the unneeded events from the queue:
/*
* Check if this file is stdin, and if so, if it is a console.
*/
if (h == GetStdHandle(STD_INPUT_HANDLE) &&
PeekConsoleInput(h, &record, 1, &num_read) == 1) {
/*
* Handle the input console buffer differently,
* since it can signal on other events like
* window and mouse, but read can still block.
*/
if (record.EventType == KEY_EVENT &&
record.Event.KeyEvent.bKeyDown) {
rc |= POLLIN;
} else {
/*
* Flush non-character events from the
* console buffer.
*/
ReadConsoleInput(h, &record, 1, &num_read);
}
} else {
rc |= POLLIN;
}
You can view the whole lovely thing here complete with comments and rationales. I'm surprised that a function like this is not more commonly available in other programs.
Problem 4: Header files
It should come as no surprise that the standard headers for Windows are also different from POSIX systems. On Windows, the standard headers seem to be optimized for precompiled header usage, and thus include the kitchen sink. In my original example, the header section looked like this:
#ifdef _WIN32
#include <ws2tcpip.h>
#else
#include <sys/socket.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <unistd.h>
#endif
With the #ifdef abolition, and because I'd rather spend 10 minutes doing a 1 minute job automatically, I added aliases for all of the missing POSIX headers that do the correct thing. The magic headers for sys/ioctl.h looks like this:
#ifndef _WIN32
#include_next <sys/ioctl.h>
#else
#include <win32netcompat.h>
#define ioctl(fd, type, arg) ioctlsocket(fd, type, arg)
#endif
The build system then puts these include files higher in priority in the compiler's include path. The magic #include_next directive tells the compiler to look for the next file in the search path, making this a no-op on a system with a system-wide <sys/ioctl.h>. After a few dozen of these, the porting job was complete. This saved rework on dozens more .c files that otherwise would have had ugly Windows-specific header definitions.
Conclusion: Monkey Patching beats a Million #ifdefs
To port LibreSSL to Windows, I essentially monkey patched Windows into both working, and failing, more like a POSIX operating system. This probably won't satisfy a POSIX conformance test, but it does automatically catch problems like close vs closesocket and maps errors consistently. The code does not need to deal with 2 different error systems, and any differences are fixed up one time in the compatibility layer. More importantly, it does this mostly through header and preprocessor magic, without adding a heavy-weight layer like Cygwin or needing to pepper the source code with #ifdefs.
Having recently found some error handling, #ifdef-related bugs in Meterpreter, it will be interesting to apply similar portability techniques there as well.
