Writing /var/www/st/gaa/wilmer/834root/wiki/data/meta/toolchain.meta failed

Building a MIPS32 toolchain

Getting a suitable toolchain for this machine was somewhat complicated for me. I'm not quite interested in building compilers from scratch and would rather abandon computers completely than to switch to Gentoo, but in this case I really had to do this.

There's buildroot that should make this kind of things easier, and it indeed can create working binaries, but there are some compatibility issues. First of all, to compile a kernel that works with the existing (binary-only) modules, it's best to use the same GCC version used for the original kernel, 3.4.2. Buildroot doesn't support this anymore. Also, current versions of buildroot can only build toolchains based on recent uClibc (0.9.29 or newer IIRC). Since uClibc doesn't really have backward compatibility as one of their goals, just replacing the uClibc on the router filesystem with a newer version is asking for problems, since you won't be able to recompile all binaries.

So if you want to build binaries for your router without having to put two different libc versions in /lib, feel free to use my notes here.

First, download these:

Don't download uClibc, it's already included in the Netgear source tree.

To get started, set up some environment variables.

export TARGET=mips-linux-uclibc
export PREFIX=/usr/$TARGET
export PATH=$PATH:$PREFIX/bin
mkdir $PREFIX

You can just do this all as root, but I did it under my usual UID and chowned $PREFIX to myself during the build process. I initially set TARGET to mips-uclibc-linux, but gcc didn't recognize this properly as uClibc. The result: It tried to find the dynamic linker in /lib/ld-linux.so.2 (IIRC) instead of /lib/ld-uClibc.so.0. So don't make this mistake (although fixing it seems as simple as creating a symlink).

To compile binutils:

./configure --target=$TARGET --prefix=$PREFIX --with-sysroot=$PREFIX --disable-nls
make install

This may bomb out on your TARGET string. To fix this, I changed configure.tgt to recognize the TARGET string without -gnu at the end. Since you're going to make a uClibc/BusyBox/Linux system with very little GNU/ involved, binutils will just have to accept this. :-P

Now, to get a good C compiler you'll have to compile it twice. It has to be tied to your libc, but to get a libc you need a C compiler first. A nice chicken-egg problem. In this case, the solution is to create a basic gcc that doesn't understand how to deal with libc.

./configure --target=$TARGET --prefix=$PREFIX --disable-nls \
  --enable-languages=c --with-gnu-ld --with-gnu-as --disable-shared --without-headers \
make all-gcc
make install-gcc

Now you have a compiler that can't compile anything that depends on libc. IOW, all you can compile is libc and a Linux kernel. If that's all you need, you're done. Otherwise, continue:

make menuconfig (if you want to change anything, although certain changes may break binary compatibility AFAIK)
PREFIX= make install

The PREFIX= part is important, since uClibc for some reason uses it for what other people use DESTDIR for. IOW, you'd end up with stuff installed in $TARGET/$TARGET/…

Now, recompile gcc. Do you feel like a Gentoo user yet? :-/ Before you start, there's one issue I decided to fix here. Newer versions of gcc already do this, but 3.4.2 doesn't: When compiling stuff for your MIPS board, you obviously want to link against MIPS libraries, not your native stuff in /usr/lib. Also, you should avoid including any files from /usr/include since they're likely to be incompatible with the router as well (think of little pieces of assembly code). If you see errors like this:

In file included from /usr/include/sys/socket.h:35,
from netlib.c:5:
/usr/include/bits/socket.h: In function `__cmsg_nxthdr':
/usr/include/bits/socket.h:271: warning: cast increases required alignment of ta rget type
/tmp/ccb6QiPR.s: Assembler messages:
/tmp/ccb6QiPR.s:88: Error: unrecognized opcode `rorw $8,$2'
/tmp/ccb6QiPR.s:93: Error: unrecognized opcode `rorw $8,$2'
/tmp/ccb6QiPR.s:332: Error: unrecognized opcode `rorw $8,$2'

This is a good sign that you were accidentally using include files from your own system while cross compiling. You could manually change all Makefiles to pass -I/usr/mips-linux-uclibc/include or you could just fix your gcc to ignore /usr/include completely. From what I understand, recent versions of gcc take care of this already when they're built as a cross-compiler, but gcc-3.4.2 unfortunately doesn't do this yet. I ended up with a gross hack, editing gcc/cppdefaults.c. This file has a list of include directories cpp should use. What I did is insert /usr/mips-linux-uclibc/include right before /usr/include:

    /* Some systems have an extra dir of include files.  */
    { SYSTEM_INCLUDE_DIR, 0, 0, 0, 1 },
    { "/usr/mips-linux-uclibc/include", STANDARD_INCLUDE_COMPONENT, 0, 0, 1 },
    /* /usr/include comes dead last.  */

This is a dirty hack and I'm sure there are better solutions, I'm open to suggestions. I tried a few solutions that seemed nicer and saner, but none of them worked properly. Anyway, with this change, you can now make your final gcc:

./configure --target=$TARGET --prefix=$PREFIX --disable-nls --with-sysroot=$PREFIX \
  --enable-languages=c --with-gnu-ld --with-gnu-as --with-headers=$PREFIX/include \ 
make install

This should all work perfectly and you should now be able to compile binaries that work perfectly on your router. You may sometimes need certain libraries. Just install them with the same target and prefix used above and you should be able to link your new binaries against them.

I also installed my recompiled uClibc version on the router because the stock one doesn't support IPv6. This was possible without breaking the old binaries (including some that seem to be compiled against uClibc 0.9.27). I haven't tried changing too many other settings since as far as I know certain changes could break ABI compatibility.

Some problems I had while doing this:

  • Even with uClibc I couldn't get the router to boot properly. Haven't tried very hard though.
  • If you have a multicore system you may be tempted to run make with the -j flag. IIRC at least gcc-3.4.2's build system doesn't have dependencies set up very well and may break when you do this. :-/
  • buildroot comes with some patches for gcc. I think none of them actually solved any of the problems I had, and most are for platforms I don't care about. I did apply them all to my source tree at some point and may have accidentally fixed a problem I don't know about.
  • IMHO, don't bother with buildroot. It's a great piece of software, but only works properly when you do things similar to what the developers do. For example, I found bugs where things break mysteriously because I want to install the toolchain in /usr and not in buildroot/staging_dir/. Also, although uClibc is still offered as an option in menuconfig, I couldn't get it to work. There were more problems, I don't remember all of them anymore. Unless you're willing to have two versions of uClibc on your router, you'll probably have to build your toolchain by hand.
  • I tried newer GCC versions, but failed miserably here as well. I suppose with some more efforts I could succeed, but didn't feel like wasting more time on it: recent versions of GCC come with a lot of extra fluff that is more complicated to build. During the build process, configure will want to compile and link some test programs (I think it's clever enough to not try to run them when building a cross compiler) and fail. IIRC I gave up on this at some point because the tests were incompatible with uClibc 0.9.28.

Anyway, these are just some random notes and memories I have from creating my toolchain. It was one of the most tedious things I've ever done, but the thing does work very well so I'm happy. :-) If you have any hints on how to do certain things better, please do tell.

toolchain.txt · Last modified: 2010/10/30 00:31 (external edit)
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki