Turning rusty tech into Rust ~ When you need to FTP but don’t want to

17 June 2021

Hannes de Jager

I believe that FTP is due for a makeover because it's so ancient that few companies want to host it, yet so many customers still want to use it. That's why fellow techies and I embarked on two open-source projects to develop a modernised FTP server for the cloud called unFTP.

In this writeup I tell you about this server and library, how you can use, customise and extend it and finally ask you to help us make it even better by contributing to its Rust codebases.

unFTP you say...?

unFTP is an open-source FTP(S) (not SFTP) server aimed at the cloud that allows bespoke extension through its pluggable authenticator, storage back-end and user detail store architectures. It aims to bring features typically needed in cloud environments like integration with proxy servers, Prometheus monitoring and shipping of structured logs while capitalizing on the memory safety and speed provided by its implementation language, Rust.

unFTP is first an embeddable library (libunftp) and second an FTPS server application (unFTP). You can run it out of the box, embed it in your app, craft your own server or build extensions for it.

unFTP tries to untangle you from old-school environments so you can move all the things, even FTP, to the cloud while your users still get that familiar FTP feeling.

Did you stop to ask if you should?

What year is it? Why are we talking about FTP?

...might as well be a comment on this blog post similar to the one I've seen on Reddit the other day. And one has to understand this sentiment because this protocol comes from the ARPANET days in the early 1970s. I mean the Internet was not even a thing yet. And yet, in this day and age, it is interesting to see how widely it is still used as a medium of business system integration or file exchange with customers. Sometimes you can't seem to escape FTP. Some of us are lucky or persistent and get our customers to move from FTP to FTPS and/or SFTP which were developed to address the growing security concerns of plaintext FTP. Most of us just make the best of it and try to touch those undiscussed servers as little as possible.

Here at bol.com - the largest online retailer in the Netherlands and Belgium and tech business employer of scores of IT people that build its seller platform - we are not able to escape FTP just yet either. We need to FTP even though we prefer not to. We also need to integrate with FTP from our microservices running on Kubernetes in the Cloud. Developers expressing their latest platform innovations in languages like Kotlin and Go are not particularly empowered by technologies like FTP. No, it's rather a millstone around the keyboard and yet, the need for data remains king. Vendors like AWS offer great solutions but still, sometimes custom business integration needs impede their use. This is one of the reasons why at bol.com we decided to develop unFTP.

OK, fair enough, how do I run it?

If you're on Linux or macOS then you can head over to the unFTP home page at github.com/bolcom/unFTP and download the binaries from there.

Chances are, though, that you would like to run this in Kubernetes. Here is an example of running unFTP in a docker container to nudge you in that direction:

docker run \
 -e ROOT_DIR=/ \
 -e UNFTP_LOG_LEVEL=info \
 -e UNFTP_FTPS_CERTS_FILE='/unftp.crt' \
 -e UNFTP_FTPS_KEY_FILE='/unftp.key' \
 -e UNFTP_SBE_TYPE=gcs \
 -e UNFTP_SBE_GCS_BUCKET=the-bucket-name \
 -e UNFTP_SBE_GCS_KEY_FILE=/key.json \
 -e UNFTP_AUTH_TYPE=json \
 -e UNFTP_AUTH_JSON_PATH='/secrets/unftp_credentials.json' \
 -e UNFTP_LOG_REDIS_HOST='redislogging.internal.io' \
 -e UNFTP_LOG_REDIS_KEY='logs-list' \
 -e UNFTP_BIND_ADDRESS='0.0.0.0:2121' \
 -e UNFTP_PASSIVE_PORTS='50000-50005' \
 -p 2121:2121 \
 -p 50000:50000 \
 -p 50001:50001 \
 -p 50002:50002 \
 -p 50003:50003 \
 -p 50004:50004 \
 -p 50005:50005 \
 -p 8080:8080 \
 -ti \
 bolcom/unftp:v0.12.10-alpine

Ignore for a moment that this example won't exactly work since you would rather specify this in a Kubernetes yaml file but this should give you an idea if you look at the configured environment variables. The above configuration will spin up an FTPS server that serves data blobs in the root of the Google Cloud Storage bucket named the-bucket-name to FTP clients as files. It binds to all addresses and listen for control connections on port 2121 as specified by UNFTP_BIND_ADDRESS='0.0.0.0:2121'. It also listens for active data connections on ports between 50000 and 50005 (UNFTP_PASSIVE_PORTS='50000-50005'). TLS communication is enabled by UNFTP_FTPS_CERTS_FILE and UNFTP_FTPS_KEY_FILE while authentication mechanisms are defined in a file pointed to by UNFTP_AUTH_JSON_PATH.

From the above example you can extrapolate how one can deploy to Google Cloud Platform and:

run unFTP in a Kubernetes pod
store files in a Google Cloud Storage bucket
get TLS certificates from Let's Encrypt via Google Cert-Manager
authenticate against encrypted credentials in a JSON payload sourced from a Kubernetes Secret
have a Prometheus instance in our environment scrape metrics from a the /metrics endpoint running on port 8080
ship logging to a Redis instance from where it's picked up by Logstash to find its home in an ELK stack.

We have a couple of docker images available on Docker hub. For instance there is also an alpine image with scuttle installed if you need to run with the Istio service mesh.

Nice, I'd like to build my own

You can embed libunftp in your Rust app or create your own FTP server with libunftp in just a couple of lines of Rust code. If you've got the Rust compiler and Cargo installed then create your project with:

cargo new myftp

Next let's add the libunftp and Tokio crates to your project's dependencies in Cargo.toml:


libunftp = "0.17.4"
tokio = { version = "1", features = <"full"> }

We will also need a dependency on a storage back-end where the files will go. We'll use the file system back-end (unftp-sbe-fs) here:


...
unftp-sbe-fs = "0.1"

Finally, let's code up the server! Add the following to `src/main.rs`:

use unftp_sbe_fs::ServerExt;

#
pub async fn main() {
    let ftp_home = std::env::temp_dir();
    let server = libunftp::Server::with_fs(ftp_home)
        .greeting("Welcome to my FTP server")
        .passive_ports(50000..65535);
    
    server.listen("127.0.0.1:2121").await;
}

The above creates an FTP server that uses a file system back-end that will put and serve files from the operating system temporary directory. We set the greeting message an FTP client will see when it connects and we define the port range that the server can listen on for data connections originating from the FTP client. Lastly, we listen for control connections on port 2121 on the local host. Not that useful yet but you get the idea.

Let's proceed to run your server with cargo run and connect to localhost:2121 with your favourite FTP client. For example:

lftp -p 2121 localhost

This should allow you to upload and download files to and from your temporary directory via FTP.

I have this other authentication system...

Of course you do. If you're opting to create your own FTP server with libunftp then chances are that you will also be implementing your own storage back-end and/or authenticator. To illustrate how this is done we show you how to implement an authenticator that will always give access to the user. Start off by adding a dependency to the async-trait crate in your Cargo.toml file:


...
async-trait = "0.1.50"

Then implement the Authenticator and optionally the UserDetail trait:

use libunftp::auth::{Authenticator, AuthenticationError, UserDetail};
use async_trait::async_trait;

#
struct RandomAuthenticator;

#
impl Authenticator<RandomUser> for RandomAuthenticator {
  async fn authenticate(&self, _username: &str, _password: &str) -> Result<RandomUser, AuthenticationError> {
    Ok(RandomUser{})
  }
}

#
struct RandomUser;

impl UserDetail for RandomUser {}

impl std::fmt::Display for RandomUser {
  fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
    write!(f, "RandomUser")
  }
}

Lastly, register this authenticator with libunftp:

let server = libunftp::Server::with_authenticator(
  Box::new(move || { unftp_sbe_fs::Filesystem::new("/srv/ftp") }),
  std::sync::Arc::new(RandomAuthenticator{})
)
.greeting("Welcome to my FTP server")
.passive_ports(50000..65535);

If you don't want to allow a user access to the files then simply return an AuthenticationError from the authenticate method above.

Call for help

unFTP as it stands today provides a minimal viable product for certain use cases but is nowhere near the one stop solution for FTP that it can become. To accomplish this a village of contributors are needed. A secret hope with this blog post is that we inspire warriors that will pick a fight with the Rust borrow checker and bring a unFTP ecosystem to life that we can be proud of, and, dare I say, perhaps even to the point where people may want to FTP even though they don't need to :-).

We would love to see many unftp-* results returned when a search is done on crates.io, the Rust package registry. Results ranging from the obviously needed to the super creative. We imagine a state where FTP integrators can bring their very custom needs to crates.io and have a shopping list to pick from:

Authenticator implementations like:

unftp-auth-ldap
unftp-auth0

User detail stores like:

unftp-usr-postgres
unftp-usr-mysql
unftp-usr-auth0

Storage back-end implementations like:

unftp-sbe-s3
unftp-sbe-azure-blobs
unftp-sbe-chrooted-fs
unftp-sbe-dropbox
unftp-sbe-gmail

The project also needs contributors to its core because what's the use of a lot of extensions without a solid core product:

Experts in FTP or ardent RFC readers to extend on the provided FTP protocol command implementations.
Experts in distributed computing to help build a truly scalable and highly available FTP solution for the cloud.
People helping with testing and benchmarking.
Implementors for things like event notification to cloud pubsub, a possible new extension point.
Scriptability through Lua.
Security experts to help ensure libunftp is as secure as it can be.

unFTP also needs to be available. So the project also needs:

Developers experienced in packaging for deployment: .deb, .rpm, archlinux
A Homebrew formula.Support on more platforms: Windows, Arm?.

And last but not least, good documentation with pleasant pictures.

Summary

In this post we explored the libunftp crate for Rust and its companion server project unFTP that was birthed by the often found need to run FTP even though it would be much rather avoided. We touched on how unFTP is aimed at solving custom FTP integration challenges in today's cloud environments. Then we gave an introduction of running the server and how to use the library and invited our readers to get their hands dirty with Rust by contributing to the unFTP project. We hope to meet some of you in the near future even though Covid-19 won't allow us to shake hands yet.

Hannes de Jager

All articles by me