T* About

trivelop

{:ok, "Let's ship it ..."}

Architecting Flow in Elixir Programs: An Introduction

Getting the “flow” of a program is one of the first things I do when looking at open source software or joining a new project at work.

The easier it is to grasp how data flows through a program, the easier it is for me as a developer to estimate the impact of changes (and the business of software is all about changes; while “writing hot new cool stuff” is most certainly fun, most professional work I’ve done falls into the “maintaining once-hot-new-cool-stuff and adapt it to changed requirements” category).

Now, writing CLIs (command line interfaces) is a personal pet peeve of mine. These are little, mostly straight-forward programs, sometimes fulfilling a single purpose (like cat or touch) and sometimes being the fascade for a host of features (like git, docker or mix).

In any case, CLIs are a perfect example to demonstrate what I would consider good “flow” (and, naturally, the ideas presented in this post are just as applicable to embedded software, web applications or any other program).

Let’s see some code

Imagine a CLI for converting images. The flow might look something like this:

Implemented as a Mix task, it could look something like this:

defmodule Mix.Tasks.ConvertImages do
  use Mix.Task

  def run(argv) do
    {opts, args, _invalid} =
      OptionParser.parse(argv, switches: [target_dir: :string, format: :string])

    glob = List.first(args) || "./image_uploads/*"
    filenames = Path.wildcard(glob)

    target_dir = opts[:target_dir] || "./tmp"
    File.mkdir_p!(target_dir)

    format = opts[:format] || "jpg"
    # TODO: ensure valid format

    results =
      Enum.map(filenames, fn filename ->
        Converter.convert_image(filename, target_dir, format)
      end)

    IO.puts("Wrote #{Enum.count(results)} images to #{target_dir}.")
  end
end

# NOTE: we will omit the definition of `Converter.convert_image/3` for now and
#       assume it works as one would expect (take a filename, convert its
#       contents to the given format and write the result to the given target
#       directory).

I have written this kind of program a dozen times before. There’s nothing really wrong with it, except that writing these kinds of throw-away scripts is much more fun than inheriting them. So let’s do our successor (or our future-self) a favor …

We could group related work inside the function if we want to illustrate the flow of our program:

defmodule Mix.Tasks.ConvertImages2 do
  use Mix.Task

  def run(argv) do
    # 1 - parse options
    {opts, args, _invalid} =
      OptionParser.parse(argv, switches: [target_dir: :string, format: :string])

    glob = List.first(args) || "./image_uploads/*"
    target_dir = opts[:target_dir] || "./tmp"
    format = opts[:format] || "jpg"

    # 2 - validate options
    filenames = Path.wildcard(glob)

    if Enum.empty?(filenames) do
      raise "No images found."
    end

    unless Enum.member?(~w[jpg png], format) do
      raise "Unrecognized format: #{format}"
    end

    # 3 - prepare conversion
    File.mkdir_p!(target_dir)

    # 4 - convert images and write them to target directory
    results =
      Enum.map(filenames, fn filename ->
        Converter.convert_image(filename, target_dir, format)
      end)

    # 5 - report results to STDOUT
    IO.puts("Wrote #{Enum.count(results)} images to #{target_dir}.")
  end
end

But this might just be a case of “commenting not-so-ideal code”, so let’s put these sections into separate functions:

defmodule Mix.Tasks.ConvertImages3 do
  use Mix.Task

  @default_glob "./image_uploads/*"
  @default_target_dir "./tmp"
  @default_format "jpg"

  def run(argv) do
    {glob, target_dir, format} = parse_options(argv)

    validate_options(glob, format)

    filenames = prepare_conversion(glob, target_dir)
    results = convert_images(filenames, target_dir, format)

    report_results(results, target_dir)
  end

  defp parse_options(argv) do
    {opts, args, _invalid} =
      OptionParser.parse(argv, switches: [target_dir: :string, format: :string])

    glob = List.first(args) || @default_glob
    target_dir = opts[:target_dir] || @default_target_dir
    format = opts[:format] || @default_format

    {glob, target_dir, format}
  end

  defp validate_options(glob, format) do
    filenames = Path.wildcard(glob)

    if Enum.empty?(filenames) do
      raise "No images found."
    end

    unless Enum.member?(~w[jpg png], format) do
      raise "Unrecognized format: #{format}"
    end
  end

  defp prepare_conversion(glob, target_dir) do
    File.mkdir_p!(target_dir)

    Path.wildcard(glob)
  end

  defp convert_images(filenames, target_dir, format) do
    Enum.map(filenames, fn filename ->
      Converter.convert_image(filename, target_dir, format)
    end)
  end

  defp report_results(results, target_dir) do
    IO.puts("Wrote #{Enum.count(results)} images to #{target_dir}.")
  end
end

It’s getting easier to see what is happening and what phases the program walks through to reach its goal.

If we revisit our diagram from the top, we start to see that we added the activity of validating our inputs:

See below how we can adapt this diagram into code using Elixir’s pipe operator (|>).

defmodule Mix.Tasks.ConvertImages4 do
  use Mix.Task

  @default_glob "./image_uploads/*"
  @default_target_dir "./tmp"
  @default_format "jpg"

  # NOTE: we could also have refactored this using `with`, but
  #       it doesn't really matter for the point I'm trying to make ^_^

  def run(argv) do
    argv
    |> parse_options()
    |> validate_options()
    |> prepare_conversion()
    |> convert_images()
    |> report_results()
  end

  defp parse_options(argv) do
    {opts, args, _invalid} =
      OptionParser.parse(argv, switches: [target_dir: :string, format: :string])

    glob = List.first(args) || @default_glob
    target_dir = opts[:target_dir] || @default_target_dir
    format = opts[:format] || @default_format

    {glob, target_dir, format}
  end

  defp validate_options({glob, target_dir, format}) do
    filenames = Path.wildcard(glob)

    if Enum.empty?(filenames) do
      raise "No images found."
    end

    unless Enum.member?(~w[jpg png], format) do
      raise "Unrecognized format: #{format}"
    end

    {glob, target_dir, format}
  end

  defp prepare_conversion({glob, target_dir, format}) do
    File.mkdir_p!(target_dir)

    filenames = Path.wildcard(glob)

    {filenames, target_dir, format}
  end

  defp convert_images({filenames, target_dir, format}) do
    results =
      Enum.map(filenames, fn filename ->
        Converter.convert_image(filename, target_dir, format)
      end)

    {results, target_dir}
  end

  defp report_results({results, target_dir}) do
    IO.puts("Wrote #{Enum.count(results)} images to #{target_dir}.")
  end
end

Use Your Imagination

The example is meant to be easily accessible and relatable.

Please imagine the presented solution for a complex app or just a non-trivial use-case for the program above, like converting the images to match certain dimensions based on their filenames, reporting errors, adding a verbose flag to give more information to the user during runtime, optionally writing the EXIF information of the original images to an external datastore and/or serving the collected image metadata through the same CLI while lazily converting new images any time the program is run.

Now we’re talking.

Requirements change, software needs to be maintained

One thing that immediately stands out in the examples above: the last version is the most verbose one (the first version was 24 lines, the last one clocks in at 62 lines).

If we assume a more complex example, this difference in lines will become less significant. In these cases, we will reap the benefits of having a more approachable codebase, cleaner stacktraces and an easier time to delete old and add new code.

The last point is paramount because requirements for software change all the time. So we want to make our programs as adaptable to change as possible.

A clear flow can enable just that. 👍

Looking back at 2017

I think it’s save to say that 2017 has been the most eventful year of my adult life.

Leaving Neopoly after 16 years of being with the company was one of the biggest decisions I ever made. It’s always hard to move on, but this has been a very special goodbye. On January 2nd, I started working at 5Minds.

In February, I finally handed in my dissertation. After 2 years of preparation and 4 years of researching and writing, this was a very special moment, parting with 400 pages that changed my life.

I’ve always been more interested in the practical side of things, both with computer science and economics. But being more interested meant a 60/40 split, where I am still very much an academic at heart. Being able to go to university next to my day-job and, over the course of 7 years, get a degree has always been a privilege to me (I got my Diploma in 2011). I feel blessed that I subsequently got the chance to work on a research project of my choosing to get a PhD as well.

ElixirConf.EU was a blast this year. It’s been a great experience, rekindling old Elixir friendships and forming new ones. As always, it’s been great to meet people from the community.

It made me tremendously happy to see how the community embraces the same principles and I am still proud to contribute something back by maintaining Credo, ElixirStatus and ElixirWeekly

In June, I had the honour to marry my partner-in-crime, Julia ❤️

In July, I took the final oral exam for completing my PhD. As I am writing this, I still haven’t fully proccessed the fact that my formal higher education, a journey I started when I was 20 years old, has come to a close.

In October, got the opportunity to represent 5Minds at this year’s RuhrJS.

The year came to a close in December with a very special request from Argentina: I became a SpawnFest judge.

I am with 5Minds for a whole year now, getting used to my new role as a software architect and teamlead. Leading teams of developers, instead of writing code full-time myself, is a great challenge. And I mean “great” both in terms of size and feeling of accomplishment.

Do I miss writing code? Sure. But working with people next to bits and bytes is a new and exciting task. This job finally combines my previous engagements - being a programmer, designer, part-time student, later part-time teacher, public speaker and author - all into one.

Also, I am going from speaking to organising: My colleague Marc and I are organising a .NET Core user group, .NET Developers Ruhr, which will meet in February for the first time.

In conclusion: What a year. I am exited for the next one! 👏

ElixirStatus is 2 years old!

ElixirStatus is two years old now. 🎁 🎉

This was my first “complete” Phoenix project, and while it is a trivial app, I am still amazed how it keeps running, humming away like an old diesel engine, doing its job for those who find it useful.

When ElixirWeekly celebrated its first birthday a couple of weeks ago, I wrote:

The Elixir community is growing and, while some would like to see an explosion in popularity akin to JavaScript, I really like to watch the steady but unstoppable progress we are achieving together.

All the tools are maturing, getting better, gaining focus and, while all the numbers are growing, the community is staying healthy and solving real problems in the real world using our favorite programming language.

Wow. It’s been a whole year. Time has flown.

It’d like to emphazise the same for ElixirStatus and give some numbers as well:

  • Twitter followers have grown to > 4,000 Elixir programmers
  • There have been 700 postings to the site in 2016/17 (vs. 600 last season)
  • With 100 new first time authors of posts

Another observation: Lots of people are subscribing on Twitter and then unsubscribe a week later. The dense information flow ElixirStatus has to offer is special in that you get a raw feed of new projects and ideas from the community. It is understandable that this kind of tweet stream can be overwhelming, annoying even.

ElixirStatus is not for everyone and that is ay-okay! But despite this quirkiness, the community still managed to double its reach. This is great, because this means the interest in Elixir is growing.

Not with the kind of stratospheric JavaScript-like growth some had hoped for, but rather a healthy, sustainable rate to grow the Elixir community around the right ideas and problems.

I’m proud to be a part of this.

Wow. It’s been two years already. Time has flown indeed.