Building file uploads

Last week, I predominantly spent my time implementing file uploads. This was a tricker feature implement on the frontend because Elm does not have good support right now for dealing with file objects. It turned out pretty slick!


Here were my essential requirements:

⇒ Users should be able to drag-and-drop files and paste files from the clipboard onto any post or reply composer
⇒ Users should see a list of attached files on posts
⇒ Images should be embedded, and files should be hyperlinked at the current cursor position in the editor

The backend #

One of my favorite characteristics of the Elixir ecosystem (and perhaps the functional paradigm in general) is the lack of “magic” that plagues the Ruby world. In general, I’ve become pretty skeptical about adding unnecessary dependencies to Level and maintain a pretty high threshold before bringing in any outside packages.

I found Arc when I began looking around for file upload solutions, which was very reminiscent of Ruby gems like Carrierwave and Paperclip. I have no doubt it’s a well-implemented library, but I desired:

⇒ To have a good understanding of what’s happening from the point that the file bits hit the web server to when they reach their final destination,
⇒ To be able to stub out the functions that made network calls to my storage provider and write tests up to that boundary,
⇒ To fully understand the different failure modes and control the retry logic,
⇒ And, to have clear, readable function pipelines in my context module representing what’s happening.

So, I decided to try working directly with the ex_aws library to implement an S3 storage adapter for file uploads. It was remarkably simple.

My public API looks like this:

# lib/level/asset_store.ex

@doc """
Uploads a file.
@spec persist_file(String.t(), String.t(), binary()) :: {:ok, String.t()} | {:error, any()}
def persist_file(unique_id, filename, binary_data) do
  |> build_file_path(filename)
  |> @adapter.persist(@bucket, binary_data)

And the S3 adapter looks like this:

# lib/level/asset_store/s3_adapter.ex

defmodule Level.AssetStore.S3Adapter do
  alias ExAws.S3

  @behaviour Level.AssetStore.Adapter

  @impl true
  def persist(pathname, bucket, data) do
    |> S3.put_object(pathname, data, [{:acl, :public_read}])
    |> ExAws.request()
    |> handle_request(pathname)

  defp handle_request({:ok, _}, pathname), do: {:ok, pathname}
  defp handle_request(err, _filename), do: err

The frontend #

Since Elm does not support file data, I had a few options: use ports to send events back and forth from the Javascript realm or implement a custom element to set up the proper event listeners and propagate custom events that Elm is able to listen for. I decided to go with the latter.

The custom element is responsible for:

⇒ Binding event listeners for drag* and paste events when its connected to the DOM
⇒ Intercepting dropped file data and making API calls to upload the files
⇒ Dispatching custom events at relevant points that can be listened for in Elm: fileAdded, fileUploadProgress, fileUploaded, fileUploadFailed, etc.

Source for the custom element is publicly available here.

Elm is responsible for:

⇒ Storing a representation of the file objects (except for the raw binary file data, which is kept in the Javascript realm) with the current state (staged, uploading, uploaded, upload failed)
⇒ Rendering a representation of file state (e.g., a progress bar while uploading)

The custom element instantiation looks like this in Elm:

type alias ViewConfig msg =
    { spaceId : Id
    , onFileAdded : File -> msg
    , onFileUploadProgress : Id -> Int -> msg
    , onFileUploaded : Id -> Id -> String -> msg
    , onFileUploadError : Id -> msg

wrapper : ViewConfig msg -> List (Html msg) -> Html msg
wrapper config children =
    Html.node "post-editor"
        [ property "spaceId" (Id.encoder config.spaceId)
        , on "fileAdded" <|
                ( [ "detail" ] File.decoder)
        , on "fileUploadProgress" <|
            Decode.map2 config.onFileUploadProgress
                ( [ "detail", "clientId" ] Id.decoder)
                ( [ "detail", "percentage" ]
        , on "fileUploaded" <|
            Decode.map3 config.onFileUploaded
                ( [ "detail", "clientId" ] Id.decoder)
                ( [ "detail", "id" ] Id.decoder)
                ( [ "detail", "url" ] Decode.string)
        , on "fileUploadError" <|
                ( [ "detail", "clientId" ] Id.decoder)

Each page that includes a post editor must implement an update handler for each of the relevant custom events and make model updates accordingly.

Overall, I was quite impressed with how far modern browsers have come implementing web standards. This would have been a much hairier endeavor just a few years ago (and probably would have required Flash components to get decent browser compatibility 🤪).


Now read this

Moving to source-available licensing

I’ve made the decision to update Level’s licensing to a source-availability scheme. This is something I’ve been giving careful thought over past few months as I evaluate my goals and my vision for the product. Ultimately, I believe it is... Continue →