Logo Computer scientist,
engineer, and educator
• Software • The KBOX project

Using standard Linux utilities in a stock, non-rooted Android device

Native implementation of vim running on an stock, non-rooted Motorola tablet
IMPORTANT NOTE for KBOX users: This page is for background information about how the non-root Linux distribution is bootstrapped onto the Android device. It no longer describes how to install KBOX manually, or even how KBOX2 system actually works internally. Instructions on installing KBOX2 can be found here; the technical background to KBOX2 is explained here.

This article follows on from my earlier one on building and running native applications on Android, and extends the method described there to install busybox and a number of common, useful Linux utilities on a stock, non-rooted Android device. My main intention with this project, which I'm calling kbox ('Kevin's implementation of busybox'), is to be able to use a terminal emulator to do rudimentary administration tasks from the prompt, just as I can on a Linux desktop, and with no more fuss and bother. I want to be able to copy, move, compress and decompress files; telnet and ssh to remote systems; use rysnc to copy files over wifi; edit text files with vim — the usual things that Linux desktop users take for granted, and which are completely lacking in Android.

One source of fuss and bother I particularly don't want is that of 'rooting' my Android devices, and the method I'm describing here is (I believe) completely non-invasive. All changes can be reversed simply by uninstalling the terminal emulator, which will take all this stuff with it.

Of course there will be some consequences of running services like sshd as a non-root user — most obviously we won't be able to use the default ports, because they are privileged. Some Linux utilities have to be modified to work in the way I'm describing, because they have hard-coded file locations (like /bin/sh and /etc/passwd, neither of which exist in Android.

Just about anything that runs on Linux probably could — with some effort — be made to work using the 'kbox' method. You could probably even build an X server and run X applications. But there's a limit to how far it's sensible to go with this — I'm suggesting adding some relatively simple console applications to what is already a perfectly workable operating system, for its stated purposes. If you need Gnome, you'd probably do better to look at one of the projects aiming to port a complete desktop Linux to Andoid hardware.

Please note

I've tested the procedure I'm describing here on a small number of Android 3.0 devices. There's no particular reason to think it won't work on others — the method is only an extension of Google's documented procedure for running native code on Android. However, file locations might be different, as might the methods available for getting files on an off the device.

I'm assuming a fairly high level of Linux knowledge, because I won't be able to describe the whole process in terms of step-by-step commands to type. If you don't know what 'unpack archive X into directory Y' means then none of what follows will make much sense.

My procedure potentially uses a lot of internal storage. SD cards can't be used, for reasons that will be explained. The procedure may be unsuitable for devices with very limited storage.

Basic principle — how to pretend to be Linux

As I described in the previous article, I'm using Jack Palevich's Android terminal emulator as my point of access to the command shell. There are other terminals, and I presume they will all work much the same way, but Mr Palevich's has the advantage of being free of charge.

To sum up the previous article, installing the terminal emulator creates a directory /data/data/jackpal.androidterm/shared_prefs, which is writeable by the emulator (and any code it runs), and on whose files execute permission can be set. So far as I know, this directory is the only place on a non-rooted device where these conditions hold good. You can't, for example, set execute permission on files on the SD card (either internal or external).

What I'm doing, in effect, is to create a small Linux distribution rooted at the directory /data/data/jackpal.androidterm/shared_prefs/kbox. Under here will be the usual bin, usr, and so on.

The most obvious challenge with this approach is finding a way to tell Linux utilities where files are. We can set $PATH to /data/data/jackpal.androidterm/shared_prefs/kbox/bin, which will go some way towards allowing executables to be located. But this directory is only applicable to installation with one particular terminal emulator — ideally we need something more generic. And that still leaves the problem of finding common directories and files like /etc/terminfo and /etc/rsyncd.conf. If these file locations are hard-coded into the program and cannot be overridden, then really there is no solution than to modify the code. However, I've tried to avoid doing this except where it is absolutely essential. Most Linux utilities can have their supporting file locations specified either as command-line arguments or as environment variables. In these cases, I have prefered to create 'wrapper' scripts for the executables, with the scripts creating the appropriate environment. This process can be made more-or-less invisible to the user.

It follows from the above reasoning that most of the wrapper scripts are going to need to know where the root of the 'distribution' is. The way I have implemented this is to use an environment variable $KBOX to point to the root of the installation, and which all other scripts (and some modified applications) use to find their own supporting files, relative to that point.

Of course, we don't want the shell user to have to set all this up at the start of each session. So I've implemented in C a wrapper (kbox_shell) around the ordinary busybox bash shell, which runs bash with the environment ready-set up. In fact, because binaries are not configurable by the end user, I've done the minimum of this work in the C program — the bulk is done in a bash script called start_bash.sh.

So the sequence of operations for starting a shell session is as follows.

  1. Run kbox_shell. This parses the full path name of kbox_shell to work out the top level directory of the installation, then sets and exports the variables $KBOX, $PATH, $HOME, $LOGNAME, $LD_LIBRARY_PATH, etc., as based on the installation directory.
  2. kbox_shell executes busbox bash with start_bash.sh as an argument.
  3. start_bash.sh sources $KBOX/etc/profile for global settings, does some further initialization, and then exec's an interactive shell.
  4. With the environment now set up, further processing by bash is as normal
  5. .
A particular word needs to be said about $HOME. Many Linux utilties need $HOME to be set, and to point to a writable file area. Trying to keep things as Linux-like as possible, the kbox scheme uses a directory $KBOX/home/[username] for the home directory. However, Android is not a multi-user system, and there will only ever be one subdirectory under home. This directory will be named according to the Android user name assigned to the terminal emulator application, and will be arbitrary (something like 'app_62'). We could call the user 'the_user', or 'me', or something. But 'app_62' is what the Android C runtime says, so code compiled with the official Android NDK will produce a compatible user name if asked to do so.

start_bash.sh does not set all the environment variables that a desktop Linux shell might use, but it's easy enough to derive new ones if necessary, based of the set that are defined, by editing start_bash.sh.

Installation procedure

If you don't want to compile any code yourself, you can just download the binary package (10Mb). Ultimately this archive could just be exploded into the relevant directory (.../shared_prefs/), but it's not that simple. The problem is that the stock Android has no tools to uncompress archives, and there are just too many files to copy them on one by one. We can't even unpack onto a directory on the SD card and then do a bulk copy, because the all-important file permissions would be lost.

So we have to follow a multi-step procedure — first installing the busybox binary, then configuring it to provide the utilities tar, gunzip, etc., then using those utilities to install the rest of the archive.

The steps I'm describing can all be carried out by entering commands into the Android keypad. With an external keyboard or a 'proper' on-screen keyboard that's quite straightforward. By a 'proper' on-screen keyboard I mean one that provides 'tab' and arrow keys — these are missing from the stock keyboard.

Alternatively, you can install a telnet daemon (as I explained in the previous article), and do the job from a desktop computer. The daemon utelnetd is included in the binary package. In the longer term, you might prefer to use ssh, but getting sshd to work under Android is not a one-file process, whereas telnet is.

1. Get the busybox binary into place

Copy the binary busybox to your device somehow. I'm assuming that incoming files end up in /data/Download. At the command prompt:
$ cd /data/data/jackpal.androidterm/shared_prefs
$ mkdir kbox
$ cd kbox
$ mkdir bin
$ cd bin
$ cat /data/Download/busybox > busybox
$ chmod 755 busybox
Note that, at this stage, you can't 'cp' from the SD card to internal storage — you have to 'cat' and then set the file mode.

You should be able to run busybox now:

$ ./busybox
and get a list of supported shell commands.

2. Configure busybox

busybox is typically linked to placeholder commands, one for each command that it supports. You could run busybox commands like this:
$ busybox ls
$ busybox cp a b
and so on, but it's not very Linux-like. But if you do
$ ./busybox ln -sf ./busybox ls
$ ./busybox ln -sf ./busybox cp
Then you can just use 'ls', 'cp', etc. Note that you need to do 'busybox ln', rather than using the version of ln that Android supplies, because the Android variant is defective.

A full build of busybox provides several hundred commands, and linking them all manually will be a chore. If you're in the same directory as the busybox binary you can do the whole job in one step, as follows:

for c in `./busybox --list`; do busybox ln -s busybox $c; done
You should now be able to run 'ordinary' commands:
$ ./ls
$ ./cp
$ ./tar
At present you won't be able to run any busybox shell commands without specifying a directory (or '.'), because the directory kbox/bin is not in the search path $PATH for the standard shell. This is one of the complications that the wrapper process kbox_shell gets around.

3. Install the rest of the bundle

Get the archive kbox.tar.gz onto the device.
$ cd /data/data/jackpal.androidterm/shared_prefs
$ gunzip /data/Dowbload/kbox.tar.gz
$ tar -k xvf /data/Download/kbox.tar
The '-k' here is so that tar will not attempt to overwrite the busybox setup created in step 2 (although it shouldn't really be a problem — the files should be the same).

You should now be able to run the busybox bash shell with the correct environment by running kbox_shell:

$ ./bin/kbox_shell

4. Configure the terminal emulator to run kbox_shell

By default the emulator will run the very limited Android shell. For convenience, it can be told to run kbox_shell instead, which will set up the busybox environment correctly (I hope) as soon as the emulator starts. There is a setting for the shell under the Settings menu. You need to put the full path in:
/data/data/jackpal.androidterm/shared_prefs/kbox/bin/kbox_shell
And, all being well, you're good to go.

Specific utilities

busybox

I have built busybox with the Android NDK, linked against the standard Android bionic library. There's a lot missing from bionic, as many developers have pointed out. But it has one big advantage over a full glibc compiler in this context — DNS name resolution is built in, and needs no end-user configuration. Building busybox with glibc seems to create a problem for DNS, because glibc requires that the DNS implementation is in a separate library and linked at runtime. And the Android linker seems to struggle with .so libraries created for glibc. In fact, the Android linker seems to struggle with everything (see below).

Not all the busybox utilities will compile with the NDK. In some cases it's obvious why — utilities to manage users don't build because the required functions simply don't exist in Android. In some cases the functions exist exist but are just stubs, in which case the utiltity will build but not work properly. I've tried to include as many of the utilities as I can get to build. In some cases, I've had to stub out part of the implementation. For example, the date utility only builds if I remove the call to sdate() (set date). Presumably Google doesn't want code to change the system time, so they haven't provided a function for it, even a stub. So the date utility will read the date, but not set it.

ncurses.so and readline.so

Many command-line utilties use these libraries, for screen control and line editing. They can, to some extent, be dynamically linked by applications, but it's not completely reliable. I've tried to build all the code in the kbox bundle to link dynamically whereever possible. For reasons I can't really fathom, sometimes it isn't possible.

ssh (client)

Provided by dropbear, unmodified build. Keys will be cached in $HOME/.ssh in the usual way. No non-standard arguments need be used, but bear in mind that you'll have to specify a specific user for the remote service because the Android user ID is arbitrary in a non-rooted device.

scp

Provided by dropbear, unmodified build. scp uses a hard-coded path to the ssh utility, which will be wrong for kbox. I have renamed the binary scp.bin, and created a script called scp that will run scp.bin with the -S switch, overriding the location of the ssh utility.

ssh (server)

This is a somewhat modified version of dropbear. The original modification was by Jakob Blomer, to account for the fact that Android has no /etc/passwd. I have further modified the program to launch kbox_shell (and therefore the busybox shell) instead of the Android shell when the remote user connects. Without this the user would be able to connect, but be left with an unworkable environment.

kbox provides a script ssh_daemon.sh that will launch the ssh server with the appropriate arguments, specifying user credentials and port number on the command line. The port number comes from $KBOX/etc/kbox_ports.sh. As always, the port number must be above 1024 for non-root usage. In addition, the user ID (specified by the -U argument) must be the user ID of the terminal emulator app running the script. ssh_daemon.sh takes care of thse issues, and can be used as a template for more appropriate customization. Note that this script sets the current username as the password, which isn't hugely secure.

The script will read an RSA server key from $HOME/.rsa_host_key. The dropbearkey utility can be used to create this key, but start_bash.sh will create it when a user logs in on console for the first time if it doesn't already exist.

(NB: 'FIXME' messages in the log output are from Google's rather incomplete implementation of the C library).

In order to connect to the sshd server run as described above, clients need to specify the port number on the command line (e.g., ssh -p 1027 [my_andoid_ip]).

rsync (client)

I am using an unmodified build of the latest rsync source, built with the Android NDK. rsync can be used as a client with the default use of ssh to start the server on the remote system. It can also be used to talk to a stand-alone rsync daemon. I haven't (yet) included an rsh client, so the use of rsh to start a remote session (-e rsh) isn't working yet.

rsync (server)

At present, rsync can only be run as a server in stand-alone daemon mode. This mode does not require any other dameon processes (e.g., sshd) to be running on the device. The potential downside is that there is no authentication of clients, so such a daemon should probably not be left running for extended periods. The command line to start the rsync daemon is:
$ rsync --daemon --config [config_file] --port [port] --no-detach \
  --log-file /dev/null
You don't need to redirect the log to /dev/null — but the standard log location won't work, so you'll have to send it somewhere. The config file specifies what files will be made available in what modes, and how they will be identified by the client. A sample config file is provided in $KBOX/etc/rsyncd.conf. This config makes all files available, read/write, under the identifier 'all'. The client will do transfers like this:
$ rsync --port [rsync_port] [device_ip]::all/path/to/file [local_path]
This is essentially the normal usage of rsync as a client, except that the path is prefixed ':all', and the port number is given explicitly.

Note that the rsync daemon will have access only to the same files as the terminal emulator has, in an unrooted device.

To simplify usage, there is a script rsync_daemon.sh that starts the daemon as above, with a port number taken from kbox_ports.sh.

vim

I am using the full version of vim, v7.3.3. It is statically linked with ncurses for simplicity. ncurses needs a TERMINFO database — at least a minimal version. In addition, it needs to be launched with command-line arguments to specify where its supporting files are. The script $KBOX/bin/vim runs the vim executable with the appropriate settings for the kbox environment. It did not have to do anything extra to get colour to work, although some people have reported that additional terminal emulator settings are needed.

bc

The underated command-line calculator utility bc builds cleanly with the NDK complier, and links readline.so for command-line editing. The built-in precompiled function library doesn't work, for reasons that are not obvious to me. That's not a big deal, since far better ones are available. I have included the general function library from phodd.net, and the bc script will load this automatically.

file

The magic number database is at $KBOX/usr/share/misc/magic.mgc. file is a script that runs the binary with the MAGIC environment variable set appropriately.

Other utilities

hexedit — builds cleanly, linked dynamically with libncurses.so.

Closing remarks

It seems that a reasonable number of common, useful command-line utilities can be made to work in a root-less Android envrionment. I was mostly interested in getting rysnc and vim running, and these work to my satisfaction, although I doubt I use more than a fraction of the capabilities of either of these utilities.

Many of the complications associated with running a non-standard filesystem layout can be hidden from the user, by careful use of scripts and occasional code changes. In fact, I was pleasantly surprised by how few changes I had to make to the code.

Some of the complications of running without root access cannot easily he hidden — the need to specify specific port numbers for services, for example. I should also point out that the aggressive power management of Android makes it quite slow to run background processes that use network connections — but that's not specifically a problem with the technique I'm describing here.

If there's any interest in any of this, it might be possible to create a self-installer that does all the setup work, perhaps bundled with a terminal emulator.

Copyright © 1994-2013 Kevin Boone. Updated May 01 2013