• Software
• The KBOX project
Using standard Linux utilities in a stock, non-rooted Android device
 | | Native implementation of vim running on an stock, non-rooted Motorola tablet |
IMPORTANT NOTE for KBOX users:
This page is for background information about how the non-root Linux
distribution is bootstrapped onto the Android device. It no longer
describes how to install KBOX manually, or even how KBOX2 system actually
works internally. Instructions on installing KBOX2 can be found
here; the technical background
to KBOX2 is explained here.
This article follows on from my earlier one on
building and running
native applications on Android, and extends the method described there to
install busybox and a number of common, useful Linux utilities on
a stock, non-rooted Android device.
My main intention with this project, which I'm calling
kbox ('Kevin's implementation of busybox'), is to be able
to use a terminal emulator to do rudimentary administration tasks from
the prompt, just as I can on a Linux desktop, and with no more fuss and
bother. I want to be able to copy, move, compress and decompress files;
telnet and ssh to remote systems; use rysnc to copy files over wifi;
edit text files with vim — the usual things that Linux desktop
users take for granted, and which are completely lacking in Android.
One source of fuss and bother I particularly don't want is that of
'rooting' my Android
devices, and the method I'm describing here is (I believe) completely
non-invasive. All changes can be reversed simply by uninstalling the
terminal emulator, which will take all this stuff with it.
Of course there will be some consequences of running services like sshd
as a non-root user — most obviously we won't be able to use the default
ports, because they are privileged. Some Linux utilities have to be modified
to work in the way I'm describing, because they have hard-coded file
locations (like /bin/sh and /etc/passwd, neither
of which exist in Android.
Just about anything that runs on Linux probably could — with some effort — be made to work using the 'kbox' method. You could probably even build an
X server and run X applications. But there's a limit to how far it's sensible
to go with this — I'm suggesting adding some relatively simple console
applications to what is already a perfectly workable operating system,
for its stated purposes. If you need Gnome, you'd probably do better to look
at one of the projects aiming to port a complete desktop Linux to
Andoid hardware.
Please note
I've tested the procedure I'm describing here on a small number of Android
3.0 devices. There's no particular reason to think it won't work on
others — the method is only an extension of Google's documented
procedure for running native code on Android. However, file locations
might be different,
as might the methods available for getting files on an off the device.
I'm assuming a fairly high level of Linux knowledge, because I won't be
able to describe the whole process in terms of step-by-step
commands to type. If you
don't know what 'unpack archive X into directory Y' means then none of
what follows will make much sense.
My procedure potentially uses a lot of internal storage. SD cards can't
be used, for reasons that will be explained. The procedure may be unsuitable
for devices with very limited storage.
Basic principle — how to pretend to be Linux
As I described in the previous article, I'm using Jack Palevich's Android
terminal emulator as my point of access to the command shell. There are
other terminals, and I presume they will all work much the same way,
but Mr Palevich's has the advantage of being free of charge.
To sum up the previous article, installing the terminal emulator creates
a directory
/data/data/jackpal.androidterm/shared_prefs, which is
writeable by the emulator (and any code it runs), and on whose files
execute permission can be set. So far as I know, this directory
is the only place on a non-rooted device where these conditions
hold good. You can't, for example, set execute permission on files on the
SD card (either internal or external).
What I'm doing, in effect, is to create a small Linux distribution rooted
at the directory
/data/data/jackpal.androidterm/shared_prefs/kbox. Under
here will be the usual bin, usr, and so on.
The most obvious challenge with this approach is finding a way to tell
Linux utilities where files are. We can set $PATH to
/data/data/jackpal.androidterm/shared_prefs/kbox/bin,
which will go some way towards allowing executables to be located. But this
directory is only applicable to installation with one particular terminal
emulator — ideally we need something more generic. And that still leaves
the problem of finding common directories and files like
/etc/terminfo and /etc/rsyncd.conf. If these
file locations are hard-coded into the program and cannot be overridden,
then really there is no solution than to modify the code. However,
I've tried to avoid doing this except where it is absolutely essential.
Most Linux utilities can have their supporting file locations specified
either as command-line arguments or as environment variables. In these
cases, I have prefered to create 'wrapper' scripts for the executables,
with the scripts creating the appropriate environment. This process can
be made more-or-less invisible to the user.
It follows from the above reasoning that most of the wrapper scripts are
going to need to know where the root of the 'distribution' is. The way I have
implemented this is to use an environment variable $KBOX to point to the
root of the installation, and which all other scripts (and some modified
applications) use to find their own supporting files, relative to that
point.
Of course, we don't want the shell user to have to set all this up at the
start of each session. So I've implemented in C a wrapper
(kbox_shell) around the
ordinary busybox bash shell, which runs bash with the environment
ready-set up. In fact, because binaries are not configurable by the
end user, I've done the minimum of this work in the C program — the
bulk is done in a bash script called start_bash.sh.
So the sequence of operations for starting a shell session is as follows.
- Run
kbox_shell. This parses the full path name of kbox_shell to work out the top level directory of the installation, then
sets and exports the variables $KBOX, $PATH, $HOME, $LOGNAME,
$LD_LIBRARY_PATH, etc., as
based on the installation directory.
kbox_shell executes busbox bash with
start_bash.sh as an argument.start_bash.sh sources $KBOX/etc/profile for global settings,
does some further initialization, and then exec's an interactive shell.
- With the environment now set up, further processing by bash is as normal
.
A particular word needs to be said about $HOME. Many Linux utilties need
$HOME to be set, and to point to a writable file area. Trying to keep things
as Linux-like as possible, the kbox scheme uses a directory
$KBOX/home/[username] for the home directory.
However, Android is not a
multi-user system, and there will only ever be one subdirectory under
home. This directory will be named according to the Android
user name assigned to the terminal emulator application, and will be
arbitrary (something like 'app_62'). We could call the user 'the_user', or
'me', or something. But 'app_62' is what the Android C runtime says, so
code compiled with the official Android NDK will produce a compatible user
name if asked to do so.
start_bash.sh does not set all the environment variables
that a desktop Linux shell might use, but it's easy enough to derive
new ones if necessary, based of the set that are defined, by editing
start_bash.sh.
Installation procedure
If you don't want to compile any code yourself, you can just
download the binary package (10Mb).
Ultimately this archive could just be exploded into the relevant directory
(.../shared_prefs/), but it's not that simple. The problem
is that the stock Android has no tools to uncompress archives, and there
are just too many files to copy them on one by one. We can't even unpack
onto a directory on the SD card and then do a bulk copy, because the all-important file permissions would be lost.
So we have to follow a multi-step procedure — first installing the busybox
binary, then configuring it to provide the utilities tar,
gunzip, etc., then using those utilities to install the
rest of the archive.
The steps I'm describing can all be carried out by entering commands into the
Android keypad. With an external keyboard or a 'proper' on-screen keyboard
that's quite straightforward. By a 'proper' on-screen keyboard I mean one
that provides 'tab' and arrow keys — these are missing from the stock
keyboard.
Alternatively, you can install a telnet daemon (as I explained in the previous
article), and do the job from a desktop computer. The daemon utelnetd is included in the binary package. In the longer term, you might prefer
to use ssh, but getting sshd to work under Android is not a one-file process,
whereas telnet is.
1. Get the busybox binary into place
Copy the binary busybox to your device somehow. I'm assuming
that incoming files end up in /data/Download.
At the command prompt:
$ cd /data/data/jackpal.androidterm/shared_prefs
$ mkdir kbox
$ cd kbox
$ mkdir bin
$ cd bin
$ cat /data/Download/busybox > busybox
$ chmod 755 busybox
Note that, at this stage, you can't 'cp' from the SD card to internal storage — you have to 'cat' and then set the file mode.
You should be able to run busybox now:
$ ./busybox
and get a list of supported shell commands.
2. Configure busybox
busybox is typically linked to placeholder commands, one for each command
that it supports. You could run busybox commands like this:
$ busybox ls
$ busybox cp a b
and so on, but it's not very Linux-like. But if you do
$ ./busybox ln -sf ./busybox ls
$ ./busybox ln -sf ./busybox cp
Then you can just use 'ls', 'cp', etc. Note that you need to do
'busybox ln', rather than using the version of ln that
Android supplies, because the Android variant is defective.
A full build of busybox provides several hundred commands, and linking
them all manually will be a chore. If you're in the same directory
as the busybox binary you can do the whole job in one step, as follows:
for c in `./busybox --list`; do busybox ln -s busybox $c; done
You should now be able to run 'ordinary' commands:
$ ./ls
$ ./cp
$ ./tar
At present you won't be able to run any busybox shell commands without
specifying a directory (or '.'), because the directory
kbox/bin is not in the search path $PATH for the standard
shell. This is one of the complications that the wrapper process
kbox_shell gets around.
3. Install the rest of the bundle
Get the archive kbox.tar.gz onto the device.
$ cd /data/data/jackpal.androidterm/shared_prefs
$ gunzip /data/Dowbload/kbox.tar.gz
$ tar -k xvf /data/Download/kbox.tar
The '-k' here is so that tar will not attempt to overwrite
the busybox setup created in step 2 (although it shouldn't really be
a problem — the files should be the same).
You should now be able to run the busybox bash shell with the correct
environment by running kbox_shell:
$ ./bin/kbox_shell
4. Configure the terminal emulator to run kbox_shell
By default the emulator will run the very limited Android shell. For
convenience, it can be told to run kbox_shell instead, which will set up
the busybox environment correctly (I hope) as soon as the emulator starts.
There is a setting for the shell under the Settings menu. You need to put
the full path in:
/data/data/jackpal.androidterm/shared_prefs/kbox/bin/kbox_shell
And, all being well, you're good to go.
Specific utilities
busybox
I have built busybox with the Android NDK, linked against the standard
Android bionic library. There's a lot missing from
bionic, as many developers have pointed out. But it has
one big advantage over a full glibc compiler in this context — DNS name resolution is built in, and needs no end-user configuration.
Building busybox with glibc seems to create a problem
for DNS, because glibc requires that the DNS
implementation is in a separate library and linked at runtime. And
the Android linker seems to struggle with .so libraries
created for glibc. In fact, the Android linker seems to
struggle with everything (see below).
Not all the busybox utilities will compile with the NDK. In some cases
it's obvious why — utilities to manage users don't build because the
required functions simply don't exist in Android. In some cases the
functions exist
exist but are
just stubs, in which case the utiltity will build but not work properly.
I've tried to include as many of the utilities as I can get to build. In
some cases, I've had to stub out part of the implementation. For example,
the date utility only builds if I remove the call
to sdate() (set date). Presumably Google doesn't want
code to change
the system time, so they haven't provided a function for it, even a stub.
So the date utility will read the date, but not set it.
ncurses.so and readline.so
Many command-line utilties use these libraries, for screen control and
line editing. They can, to some extent, be dynamically linked by applications,
but it's not completely reliable. I've tried to build all the code in
the kbox bundle to link dynamically whereever possible. For reasons
I can't really fathom, sometimes it isn't possible.
ssh (client)
Provided by dropbear, unmodified build. Keys will be cached in $HOME/.ssh in
the usual way. No non-standard arguments need be used, but bear in mind
that you'll have to specify a specific user for the remote service because
the Android user ID is arbitrary in a non-rooted device.
scp
Provided by dropbear, unmodified build. scp uses a hard-coded path to the ssh
utility, which will be wrong for kbox. I have renamed the binary scp.bin,
and created a script called scp that will run scp.bin with the -S switch,
overriding the location of the ssh utility.
ssh (server)
This is a somewhat modified version of dropbear. The original modification
was by Jakob Blomer, to account for the fact that Android has no
/etc/passwd. I have further modified the program to launch
kbox_shell (and therefore the busybox shell) instead of
the Android shell when the remote user connects. Without this the user
would be able to connect, but be left with an unworkable environment.
kbox provides a script ssh_daemon.sh that will launch the
ssh server with the appropriate arguments, specifying user credentials
and port number on the command line. The port number comes from
$KBOX/etc/kbox_ports.sh. As always, the port number must
be above 1024 for non-root usage. In addition, the user ID (specified
by the -U argument) must be the user ID of the terminal emulator
app running the script. ssh_daemon.sh takes care of thse
issues, and can be used as a template for more appropriate customization.
Note that this script sets the current username as the password, which
isn't hugely secure.
The script will read an RSA server key from $HOME/.rsa_host_key. The
dropbearkey utility can be used to create this key, but
start_bash.sh will create it when a user logs in on
console for the first time if it doesn't already exist.
(NB: 'FIXME' messages in the log output are from Google's rather
incomplete implementation of the C library).
In order to connect to the sshd server run as described above, clients
need to specify the port number on the command line (e.g.,
ssh -p 1027 [my_andoid_ip]).
rsync (client)
I am using an unmodified build of the latest rsync source, built with the
Android NDK. rsync can be used as a client with the default use of
ssh to start the server on the remote system. It can also be used to
talk to a stand-alone rsync daemon. I haven't (yet) included an rsh client,
so the use of rsh to start a remote session (-e rsh) isn't working
yet.
rsync (server)
At present, rsync can only be run as a server in stand-alone daemon mode.
This mode does not
require any other dameon processes (e.g., sshd) to be running on the device.
The potential downside is that there is no authentication of clients, so
such a daemon should probably not be left running for extended periods.
The command line to start the rsync daemon is:
$ rsync --daemon --config [config_file] --port [port] --no-detach \
--log-file /dev/null
You don't need to redirect the log to /dev/null — but the
standard log location won't work, so you'll have to send it somewhere.
The config file specifies what files will be made available in what
modes, and how they will be identified by the client. A sample config
file is provided in $KBOX/etc/rsyncd.conf. This config makes all files
available, read/write, under the identifier 'all'. The client will
do transfers like this:
$ rsync --port [rsync_port] [device_ip]::all/path/to/file [local_path]
This is essentially the normal usage of rsync as a client, except that
the path is prefixed ':all', and the port number is given explicitly.
Note that the rsync daemon will have access only to the same files as the
terminal emulator has, in an unrooted device.
To simplify usage, there is a script rsync_daemon.sh that starts
the daemon as above, with a port number taken from kbox_ports.sh.
vim
I am using the full version of vim, v7.3.3. It is statically linked
with ncurses for simplicity. ncurses needs
a TERMINFO database — at least a minimal version. In addition, it needs
to be launched with command-line arguments to specify where its
supporting files are. The script $KBOX/bin/vim runs the vim
executable with the appropriate settings for the kbox environment. It did
not have to do anything extra to get colour to work, although some people
have reported that additional terminal emulator settings are needed.
bc
The underated command-line calculator utility bc builds
cleanly with the NDK complier, and links readline.so for
command-line editing. The built-in precompiled function library doesn't
work, for reasons that are not obvious to me. That's not a big deal,
since far better ones are available. I have included the general
function library from
phodd.net,
and the bc script will load this automatically.
file
The magic number database is at $KBOX/usr/share/misc/magic.mgc.
file is a script that runs the binary with the MAGIC
environment variable set appropriately.
Other utilities
hexedit — builds cleanly, linked dynamically with libncurses.so.
Closing remarks
It seems that a reasonable number of common, useful command-line utilities can
be made to work in a root-less Android envrionment. I was mostly
interested in getting rysnc and vim running,
and these work to my satisfaction, although I doubt I use more than a fraction
of the capabilities of either of these utilities.
Many of the complications associated with running a non-standard filesystem
layout can be hidden from the user, by careful use of scripts and occasional
code changes. In fact, I was pleasantly surprised by how few changes I
had to make to the code.
Some of the complications of running without root access cannot easily he
hidden — the need to specify specific port numbers for services, for example.
I should also point out that the aggressive power management of Android makes
it quite slow to run background processes that use network connections — but that's not specifically a problem with the technique I'm describing here.
If there's any interest in any of this, it might be possible to create a
self-installer that does all the setup work, perhaps bundled with a terminal
emulator.
|
