“Maybe I’m crazy, but in those type of situations, I immediately step back and see if I did something wrong.”

I made my agreement visible by nodding vigorously. My colleague accurately described my reflex in many situations where something goes wrong or I’m being blamed.

I call it the “Am I the A**shole? Reflex” (The AITA? Reflex). There is (somewhat) famous subreddit called r/AmITheA**hole? where anyone can post about an experience where someone was being an a**hole to them and reflect on the question “…But am I the a**hole here?”. The stories range from hilarious to genuinely intriguing, sometimes resulting in thought-provoking responses that cause the poster to realize that they are, indeed, the a**hole.

My own experience with the AITA? Reflex has been an emotional rollercoaster. For a long, long time, I didn’t even think about it. Then, when I became a manager for the first time, I became concerned that this was because of a lack of self-confidence and that this instinct needed to be overcome.

With time, I’ve realized that asking, “Is this my fault?” before reacting is extremely important when leading any team or project.

It may take seconds to answer, “Most definitely, not my fault.” But if you have a hint of doubt that it could be your fault, you need to pull on that thread and start asking questions and self-reflecting.

The last thing you want to do to peers and talent is show that you can’t own up to your mistakes. If you give into the fear of admitting fault, you set a toxic example for your organization: shifting blame is the norm. You encourage a culture of covering up mistakes rather than figuring out how to prevent them.

To avoid this, I often ask myself, “Could I have done something to prevent this?” If the answer is yes, I ask, “Was doing that thing reasonable to expect of myself?”

Say something immediately if it was your fault and it was reasonable to prevent it. If you know what could have prevented it, say it. Say what you’ll do to prevent it from happening again, or say that you’ll follow up with that information soon.

In a group setting, admitting that a situation went south because of you can immediately de-escalate a tense room. At the same time, it sets an example that it’s okay to admit fault and that we, as a team, are less concerned about blame and more about learning and moving on.

When you admit fault one on one, you can have a potent conversation about the nuances of why you may have screwed up. You can use it as a learning moment to underscore what you’ll do to remedy the situation and prevent it from happening again. It can be an essential moment to bond with a peer or report and build rapport.

If I couldn’t have prevented something bad from happening, or it wasn’t reasonable to do so, I start reflecting a little deeper: )“Is it possible that others could simply perceive this as your fault even if it wasn’t? Why?”

When you’re in leadership, perception is often reality. So, knowing how your actions are perceived is just as important as the actions themselves. In these situations, it’s important not to blame someone else but to calmly give as much context as possible.

What you shouldn’t do is point the blame. Pointing the blame elsewhere is the only thing worse than not admitting fault. If after giving that context, others don’t understand that it’s not your fault, then you have the choice of agreeing to disagree or falling on the sword in order to move on. Taking the blame when it’s not your fault requires a careful cost/benefit analysis; you don’t want to hurt yourself. That’s a subject for another post…

Tips on Admitting Fault

If you struggle with admitting fault, knowing your audience is critical. For some situations and people, humor works. Sometimes, specific actions are necessary to show it won’t happen again, especially if you’re managing up. Sometimes, just being sincere is the best default.

Here’s some ideas that might help.

Simple sincerity

“I’m feeling pretty embarrassed right now. That was my fault, I didn’t [y] and should have [y], my apologies. Let me figure out how I can prevent that from happening again.”

Start humorously, pivot to sincerity

“Well, I seem to have done it again. I’ve failed you! Can you forgive me?! [… pivot to sincerity ] But, really, I am sorry, I won’t let that happen again. Can we talk about how I can help fix it?”

Specificity

“This is definitely my fault. I did not [a] and should have [b]. I’m going to figure out how to fix this, but I believe that [doing c, d, and e] will probably help. In the future, I will [f, g and h] to make sure it doesn’t happen again.”

Empathy

“This was my fault. I can tell that you’re really [upset/frustrated/insert emotion here] because of it and I’m really sorry. I’m going to try to fix it by [a, b and c]. How does that sound? Do you want to talk about it more?”

Go forth and check yourself

Especially right now, in a bad tech job market where people are tense and worried about getting laid off, we need to remember to admit fault when we screwed up and set a good example for others.

Always question your actions in a healthy way. Don’t perseverate on it or let it get in the way of taking action, but also… Don’t be an a**hole.

In this post, I’m going to outline some good practices I’ve learned along the way and (hopefully) convince you why and how you should love 🪵 logging 🪵 and why you should consider using a logging framework immediately instead of console.log statements.

I’ll also show by example what some of the most popular node logging libraries do well and how they stack up against what I’m suggesting.

I am going to be referring to code snippets and output from a sample project I put together called node-logging-examples. You can run this project over on Replit by using the “Shell” button at the bottom right.

Lessons from the Frontline - Aka “Why should I listen to this guy?”

I’ve supported several server side applications. Including being on call for those applications. When you get woken up in the middle of the night for an alert, you want to make sure that your logs are watertight, otherwise, you’ll be water logged (GET IT!?). So watertight you can read them squinting in a dreamy haze! I know what to do, and probably more importantly what not to do.

It’s also worth knowing that I spent a lot of my career in the Java world (though not recently). Since this post is targeted at Javascript developers, reserve your judgment when I say: Java does logging very well. There are several battle-tested libraries that allow for out of the box ease, as well as endless customizability.

Okay, on with the show…

Use a framework built for logging, not console.log

Table stakes - The rest of this article will hopefully convince you why! I know some people will disagree, and for really really simple apps you can get away with it, but I can’t recall a case in my career where I’ve started with simple console.log statements and didn’t regret it later.

Separate logging configuration from logging code

Whenever possible, you want the ability to dial logging levels up and down at granular levels without performing a code change. This could be via environment variables, or even through external configuration files or databases.

Either way, during a production incident, or even while debugging, being able to dial up or down logging levels with minimal risk is a lifesaver.

At a very minimal level, you can simply set a LOG_LEVEL environment variable which your logging setup inspects and sets the appropriate level.

Going to the more advanced levels, you might have an external configuration file that defines log levels. Log4js is the only framework I saw with a specific option to configure logging with an external source right out of the box.

You may be thinking: Of course, we set our log levels outside of code! However, I would ask if you can do it only for the areas of the application which need the logging tweaked?

Read on…

Configuring logging on a per file/module basis

Being able to dial up logging levels in very specific parts of your application can help reduce a lot of noise when troubleshooting. Of course, modern log ingestion/viewing tools like Elasticsearch or Splunk allow you to do really advanced filtering, but what about when you just want to trace through what’s happening for a given user in your application? It’d be nice to see just a bit of logging for the parts of your application that aren’t having issues, but a ton for the areas of concern.

A lot of node.js frameworks do not have the ability to do this out of the box, instead relying on you to roll your own configuration. Which has its drawbacks and benefits.

I’ve been programming in Node.js for years now, but still, one of the best features of Log4j (the de-facto standard Java logging library) I miss the most while programming in Node is being able to easily customize which modules are logging to what logging level.

As an example, these Java classes:

class Main {
  static final Logger logger = LogManager.getLogger(Main.class.getName());
  
  public static void main(String[] args) {
    logger.info("Begin app ...");

ExternalService.makeACall("test request info");
    FailingService.makeACall("something that will fail");
    InternalBusinessLogic.wickedImportantRules("test input");
    logger.info("Completed app!");
  }
}

public class ExternalService {
  static final Logger logger = LogManager.getLogger(ExternalService.class.getName());
  public static void makeACall(String request) {
    logger.info("Making external service call");
    logger.debug("External Service Call Request {}", request);
  }
}

public class FailingService {
  static final Logger logger = LogManager.getLogger(FailingService.class.getName());
  public static void makeACall(String request) {
    logger.info("Making failing external service call");
    logger.error("Something went wrong in the call!");
  }
}

public class InternalBusinessLogic {
  static final Logger logger = LogManager.getLogger(InternalBusinessLogic.class.getName());

  public static void wickedImportantRules(String arg) {
    logger.info("Running business rules");
    logger.debug("Fixing Johnson rod");
    logger.trace("Arguments for business logic: {}", arg);
    logger.trace("More information here that's really low level");
    logger.debug("Done running business rules");
  }
}

Will output like this on the console (configuration dependent):

22:20:44.283 [main] INFO  FailingService - Making failing external service call
22:20:44.285 [main] ERROR FailingService - Something went wrong in the call!
22:20:44.286 [main] INFO  InternalBusinessLogic - Running business rules
22:20:44.286 [main] DEBUG InternalBusinessLogic - Fixing Johnson rod
22:20:44.294 [main] TRACE InternalBusinessLogic - Arguments for business logic: test input
22:20:44.294 [main] TRACE InternalBusinessLogic - More information here that's really low level
22:20:44.294 [main] DEBUG InternalBusinessLogic - Done running business rules

You can dial logging levels and even separate outputs for individual files/classes. See this ReplIt for a working example you can run.

I haven’t found any Javascript frameworks that allow this type of customizability, out of the box, however, some logging frameworks have the concept of categories or child loggers that allow you to do something similar.

For example, I was able to get Log4js to do this with a little bit of help from their categories API.

This is the type of output you see with my approach, which results in this type of output where the file name is listed before the message.

[2023-02-13T10:58:00.168] [INFO] log4js/example-log4js.js - Syncing clients...
[2023-02-13T10:58:00.172] [DEBUG] log4js/log4jssubfolder/clientservice.js - Getting last run
[2023-02-13T10:58:00.174] [DEBUG] log4js/log4jssubfolder/clientservice.js - Getting clients updated since 2023-02-13T10:58:00
[2023-02-13T10:58:00.174] [INFO] log4js/example-log4js.js - Fetching clients updated since 2023-02-13T10:58:00
[2023-02-13T10:58:00.174] [DEBUG] log4js/log4jssubfolder/clientservice.js - Doing something really complicated
[2023-02-13T10:58:00.174] [ERROR] log4js/example-log4js.js - ERROR Calling Remote Service
[2023-02-13T10:58:00.174] [ERROR] log4js/example-log4js.js - Service Call Result []
[2023-02-13T10:58:00.174] [DEBUG] log4js/log4jssubfolder/anotherservice.js - Doing something really fun!
[2023-02-13T10:58:00.174] [DEBUG] log4js/log4jssubfolder/clientservice.js - Setting last run

And you can dial the level up and down with a glob pattern (see the categories block):

{
  "appenders": { "standard": { "type": "stdout" } },
  "categories": {
    "default": { "appenders": ["standard"], "level": "info" },
    "log4js/log4jssubfolder/**.js": {
      "appenders": ["standard"],
      "level": "debug"
    },
    "perf": { "appenders": ["standard"], "level": "trace"}
  }
}

It would be great if this were built into log4js, but it’s fairly easy to see how it’s done (source here). Here are the highlights:

import { fileURLToPath } from 'url';
import glob from 'glob';


const defaultOptions = {
  ignore : ['default'],
}

function getLog4JOptionsUnGlobbed(globbedLog4JsOptions, globOptions) {

  const mergedGlobOptions = {...defaultOptions, globOptions};
  const unglobbedOptions = JSON.parse(JSON.stringify(globbedLog4JsOptions));
  for (const category in globbedLog4JsOptions.categories ) {
    const matchingFiles = glob.sync(category, mergedGlobOptions);
    if ( matchingFiles && matchingFiles.length > 0 ) {
      matchingFiles.forEach((file) => {
        unglobbedOptions.categories[file] = globbedLog4JsOptions.categories[category];
      })
      delete unglobbedOptions.categories[category]
    }
  }
  return unglobbedOptions;
}
function createModuleLogger(fileName) {
  // you can just use __filename if using commonjs modules! (eg require() format)
  const __filename = fileURLToPath(fileName);
  const relativeModuleName = path.relative('', __filename);
  return log4js.getLogger(relativeModuleName);
}

You can create new loggers like this in each of your modules/files:

// again, you don't need import.meta.url if you're using commonjs
const logger = createModuleLogger(import.meta.url);

After that, we’re using log4js categories to add the filename for output and configuration. With a little help with some preprocessing the log4js config file with the glob library, we can dial those levels up and down on a per file (or folder) basis.

Use low-level log statements instead of comments to document code

You can use logging statements to simultaneously document your code and provide useful logging output that you can turn on or shut off with a logging framework.

Consider this simple example, ignore the details, they’re purposely opaque so you can pretend you’re in a new codebase:

const syncClientsWithoutLoggingFramework = async function() {
  console.log('Syncing clients...');
  const lastRun = await getLastRun();
  const from = lastRun || moment().subtract(3, 'days').format(DATE_FORMAT);
  const timeOfLastFetch = moment().format(DATE_FORMAT)
  const clients = await clientsUpdatedSince(timeOfLastFetch);

  console.log(`Fetching clients updated since ${timeOfLastFetch}`);
  if (clients.length === 0) {
    console.log('No updated clients.')
    return;
  }

  // call out to another service to modify the clients list
  // with some _really important business logic_
  const modifiedClients = await doSomethingReallyComplicatedInAnotherService(clients);
  if ( modifiedClients < 1 || (modifiedClients.errors && modifiedClients.errors.length > 0) ) {
    console.error('ERROR Calling Remote Service')
    console.error(`Service Call Result ${JSON.stringify(modifiedClients, null, 2)}`)
  }

  // Store the last time we updated the clients.
  await setLastRun(moment())
  return modifiedClients;
}

That results in this logging output:

Syncing clients...
Fetching clients updated since 2023-01-23T17:25:18
ERROR Calling Remote Service
Service Call Result []

Well, it’s better than nothing! But I’m still kinda confused about what’s going on.

If we add a logging framework (here I’m going with Pino) and use the lowest log level (in this case, trace), this is what the code looks like as we run it:

const logger = pino({level: 'trace'});

const syncClientsWithLoggingFramework = async function() {
  logger.info('Syncing clients...');
  const CLIENT_LAST_RUN_KEY = 'CLIENT_LAST_RUN_DATE';
  const lastRun = await getLastRun();
  const from = lastRun || moment().subtract(3, 'days').format(DATE_FORMAT);
  const timeOfLastFetch = moment().format(DATE_FORMAT)
  const clients = await clientsUpdatedSince(timeOfLastFetch);

  logger.info("Fetching clients updated since %s", timeOfLastFetch);
  logger.trace("Fetching clients with parameters from: %s and timeOfLastFetch: %s", from, timeOfLastFetch);
  if (clients.length === 0) {
    logger.warn('No updated clients.')
    return;
  }

  logger.debug("Calling out to another service to do some really complicated business logic");
  const modifiedClients = await doSomethingReallyComplicatedInAnotherService(clients);
  if ( modifiedClients < 1 || (modifiedClients.errors && modifiedClients.errors.length > 0) ) {
    logger.error('ERROR Calling Remote Service')
    logger.error(`Service Call Result ${JSON.stringify(modifiedClients, null, 2)}`)
  }
  logger.trace("Done calling out and doing important things.");

  logger.trace("Setting last run date so we know where to pick up next time.")
  await setLastRun(moment())
  return modifiedClients;
}

This is what the output looks like, and before you judge, hold up:

{"level":30,"time":1674503301467,"pid":918,"hostname":"f6572868d2b5","msg":"Syncing clients..."}
{"level":30,"time":1674503301469,"pid":918,"hostname":"f6572868d2b5","msg":"Fetching clients updated since 2023-01-23T19:48:21"}
{"level":50,"time":1674503301469,"pid":918,"hostname":"f6572868d2b5","msg":"ERROR Calling Remote Service"}
{"level":50,"time":1674503301469,"pid":918,"hostname":"f6572868d2b5","msg":"Service Call Result []"}

You might think “But when I’m developing, this looks awful!” And you’re right.

But with a logging framework, you can easily configure your logs to be prettified and colorized for development, along with dialing the logging level up to get more information:

const logger = pino(
  {
  level : 'trace',
  transport: {
    target: 'pino-pretty',
    options: {
      colorize: true
    }
  }
});

Alternatively, Pino lets you use pino-pretty as a command line utility like this:

npm start | pino-pretty

You can even put a separate pino-prettyrc config to customize the output as desired.

This is what the colorized, prettified output looks like:

With the addition of a couple of lines of configuration, you have extremely useful JSON objects for production logging, as well as nicely readable, colorized log statements for my command line development with a timestamp to know when things were logged.

Using environment variables, you can very easily drive these logging configurations for local versus production. There is an example of this with Pino in my example project here. Using console logging makes this extremely hard.

Bonus: Having a logging Adapter

This one is a nice to have, but a lot of people don’t really think about it. Let’s go back to Java for a minute to illustrate: Generally, most Java applications reach for the Log4J Api to log information. This is an interface for logging in Java. Meaning that other libraries can implement that interface. In Javascript, we don’t have interfaces, in Typescript we kinda do, but the point is to have an adapter that delegates the act of logging to a library you configure.

Why is this nice? Well, let’s say you start with a logging framework that serves you well while your app is small, but then you switch to using Elasticsearch to ingest logs. Maybe there’s another framework that provides really easy integration to ingest Elasticsearch logs. You can switch out that implementation without changing your actual code. This also makes it easy for library maintainers to have a consistent way to log output.

(As a side note, most people in the Java world just use the Log4J implementation for everything, but it’s nice to have options.)

Compare this approach against Node, where there are a number of methods for logging, but no standard interface or adapter. The best we can do is use the traditional console.log, console.error (and so on) statements and then use another tool to scrape the console output to ingest in a different way. Certainly nothing wrong with that, but it can get kinda complicated depending on your deployment setup. However, Log4js, one of the logging frameworks I’ll go over, does have an API only library that you can use.

Node Logging Frameworks - An selected overview

This is by no means an extensive list of logging frameworks for Node, but rather a specially selected list based on popularity and my own use and interests based on my recommendations above.

For another great analysis, check out this article.

Bunyan

⭐7k, 2 million weekly downloads on NPM

This library has been around since the early days of Node and is well respected. Unfortunately, the GitHub project hasn’t been updated in 2 years as of this writing, and it is also somewhat of a laggard in the performance area as you’ll see below. There is no ability to do external configuration that I see, and it would be fairly difficult to roll your own.

For this reason, I don’t recommend it.

Log4js

⭐5.6k, 3.7 million weekly downloads on NPM

I came into this article never having used this library. Probably because I thought “Why would I want to use a java-like library in node?” As the README itself states:

Although it’s got a similar name to the Java library log4j, thinking that it will behave the same way will only bring you sorrow and confusion.

While it’s a bit slow on the performance end of things (which you can see in the bonus section at the end of this article) its out of the box configuration settings are very easy to use and extremely useful. Pino also allows a huge amount of customization and much, much better performance, but since it works with streams it can be harder to reason about. Through writing this article, I became a big fan of log4js, despite its name.

LogLevel

⭐2.4k, 9.3 million weekly downloads on NPM

It’s so simple! Definitely worth a look, but I think LogLevel’s brilliance lies in having a logging framework that works with your browser’s console or your server’s console. Out of the box, you can’t even get timestamps without some gymnastics.

Pino

️⭐10.9k, 4.5million weekly downloads on NPM

This framework is newer than the others, having gone 1.0 in 2016. It has been making the rounds in Node projects at DEPT® recently to good fanfare amongst our developers. As it claims, it is fast and very customizable. To me its best feature is the ability to perform logging on a separate thread asynchronously with the flip of a configuration switch. Very cool. On any project that required some serious log gymnastics and high performance, I’d definitely go with Pino.

Winston

⭐21.1k, 12.6million weekly downloads on NPM.

One of the more established and popular options, having been around since 2011, I’ve used Winston personally in many past projects. My opinion of it before going into this analysis was that it was … fine. A little wonky to set up, and the silly logging level is cute, but leaves me scratching my head why they didn’t use trace like almost every other logging framework.

Winston allows for good customization of logging transports and the ability to customize formats quite well. Its configuration API is a bit wonky though. I’ve used it in production applications for years and never really liked it, but rather just accepted its wonkiness. Now that I know there are better options, I don’t think I’d go back to it.

My Takeaways

First, I don’t think I would ever start a project with plain console logs again after knowing what kind of benefits you can get from a logging framework over time. I know some will disagree with me, and that’s fine. But, I’d ask that you consider it. Logging can save you a lot of headaches in an application that’s operational.

For Node.js, there are lots of great logging frameworks available. If low level customizability and performance are your utmost concern, go with Pino. Its ability to take streams and log asynchronously out of the box is amazing.

For everything else, I can honestly recommend log4js. I hadn’t used it (or even heard of it!) before I started writing this, and after working with it, I really loved its ability to get up and running fast and the ease of customization. As you saw here, it’s a bit slower, but for the majority of applications I work on, this isn’t much of an issue, especially since most production logging levels are dialed way back to info and above level log output. For my next project, I plan on using log4js if I have the choice.

Go forth and Log!

What About Logging in the Frontend Client?

By Jake Rainis

Historically, it could be argued this sort of complexity on the server is far greater than on the client since the front-end has traditionally been more responsible for layout and aesthetic than it has for business logic. However, the behavioral complexity of modern front-end applications continues to increase, particularly in framework-driven front-ends such as React, Angular, or Vue. And depending on the context of how the front-end application operates functionally, logging mechanisms could provide a great deal of insight into behavior resulting in an easier way to identify and isolate bugs. \

So, where does that leave us on the front-end?

Of course, we all know that we can drop console.logs anywhere we please to debug an issue during development, but this isn’t a mature solution. These transient log statements are isolated to a single user’s browser session, and then they’re gone forever. Front-end applications run in the browser and don’t have the same luxuries as applications running on a server where events can be captured. \

What About My NextJS App?

You might be wondering about full-stack frameworks such as a Next, Remix, or Nuxt. These too run on a Node server, so they _do _enable logging solutions… But only on the server-side aspects of the application.

For example, let’s consider a Next application that uses the Pino library (Next recommends Pino). Dropping a Pino log within a Next API route, a middleware, or even in one of Next’s server helpers such as [getServerSideProps](https://nextjs.org/docs/basic-features/data-fetching/get-server-side-props) will result in the type of log we’re after here. On the other hand, dropping a Pino log into one of the application’s React components would only result in a log to the browser’s console. Once again, this is helpful for debugging throughout the development cycle, but it doesn’t help engineers when the application is running in production since the log is never captured!

Front-End Logging Solutions

As you’ve likely surmised by now, we can’t effectively capture a log from a client-side application without some additional effort. But it is entirely possible! After all, this is a common conundrum for application developers.

The gist of front-end logging lies in sending the log events somewhere to be captured and this could even be accomplished through a custom hand-rolled solution that sends events to your back-end (though, be cautious of the overhead this might introduce). However, there are also third-party solutions out there to make the job easier.

Sentry is perhaps the most battle-tested error and performance monitoring service that plugs seamlessly into any application — even front-end applications. But there are others too, like BugSnag and Logflare (recently acquired by Supabase).

One aspect of the front-end that we don’t have to worry as much about when monitoring a back-end are the browsers themselves. _What browser/version is the end-user viewing our app from? Oh jeez… what if it’s Internet Explorer? Just kidding. What device are they on? What user flow did they follow to trigger this event/error? _One one the great selling points about the third-party services listed above is that they capture this information. This additional context can make bug reproduction and squashing much easier for application engineers.

These tools will come with a price tag when used at scale, but all have generous free tiers that might be sufficient for smaller teams and applications. And since they also work for back-end technologies, logging for all portions of an application can be captured, evaluated, and triaged in one place.

Does Every Front-End Need Logging?

In general, logging and monitoring is a best practice and should not be an afterthought when it comes to a back-end application. After all, it’s simple and quick to implement.

But is it absolutely necessary on the front-end? Well, any front-end app would benefit, but the right answer is subjective. A larger and/or more complex front-end that manages a good deal of business logic would be a better candidate than a relatively static front-end that doesn’t have much complex functionality — particularly if it’s well-tested. Ultimately, it should be a discussion with the product and engineering team to determine what monitoring solutions make sense for your application. \

Speaking of, we here at DEPT® have quite a bit of experience in the realm of application monitoring. We’d love to chat more about it, so don’t hesitate to reach out!

BONUS SECTION: Logging Framework Performance Analysis

If you’ve made it this far, I appreciate you and welcome, fellow logging nerd!

Now how about performance? Pino claims to be super fast. Let’s find out!

To test, I ran a loop of 100,000 simple statements like this (with varying syntax for each logging system):

  const numberOfLoops = 100000
  let startTime = new Date();
  for(let i = 0; i <= numberOfLoops ; i++) {
    console.info(`Test Log Message ${i}`)
  }
  let endTime = new Date();

  console.info(`Time to execute ${numberOfLoops} log messages: ${(endTime-startTime) / 1000} seconds`);

To get a better sample, I ran the node process using the GNU time command 20 times each using a bash script. I purposely wrote the output to a file so my tty was not the bottleneck. This is a sample of the output of each command in the output files:

/usr/bin/time -a -o pino-times-output.txt node index.js -f pino -p >> pino-output.txt
<< snip ... >>
{"level":30,"time":1675893207318,"pid":54614,"hostname":"Matts-16-Macbook-Pro.local","msg":"Time to execute 100000 log messages: 517 ms"}
        0.62 real         0.39 user         0.15 sys

This was running on my Intel 2.6ghz 6 core i7 Macbook with 16gb of ram. I used the “real” time for comparison.

Of course, this probably isn’t a perfect test, but let’s see what happened:

And it’s console winning the race by a nose!

Let’s take a deeper look.

The Laggard: Log4js

The laggard in the race was log4js, coming in almost 200ms longer than any other framework at an average of 1.54 seconds.

The middle tier

Second to last was Bunyan, doing slightly better than Log4js at 1.44 seconds. Third to last was Winston at 1.42 seconds, then LogLevel at 1.38 seconds.

Runner Up: Pino

As promised, Pino did very well, coming in only about 80 milliseconds slower than console logging. And that’s with synchronous logging, not async logging, which offloads logging onto worker threads to make logging a non-blocking operation, which is awesome. Of course, the whole command would likely take just as long with the worker thread, but it’s nice to know that for an online application your logging won’t be blocking your application’s threads.

Console

Probably not a surprise, but good old console.log came in first by about 80 milliseconds. Given that there’s nothing else happening with the logs, this makes total sense. If performance is your concern, console.log can’t be beat! But, since it’s not nearly as customizable, there are plenty of tradeoffs to consider.

NOTE: This article was originally written for my employer, DEPT® Agency, you can find the original article on DEPT®’s engineering blog.

We’re pivoting. We need a new app, fast! New idea is basically Uber for dogs. If you need a cute dog, quick, this app will get you a dog to rent, FAST. It has to have an excellent native iOS and Android app. Need it by next monday.

After wiping up the coffee I spit out on my desk and gathered my sanity, I was blanking on where to start. Then I remembered that ChatGPT is here to help me get started.

So, I asked ChatGPT to help us get started with our app architecture. This is what happened.

Huh, well, I’m a little put off by the level of confidence, but that’s actually … Better than I expected.

I’m not just gonna sit here and let AI take over my job though, I can still do better.

A queue will definitely work, but what about “dogs” screams “queuing” to ChatGPT? I’m not quite sure why dog rentals need a queue based system, but I’m not against it either.

And more importantly, what’s the justification for using a NoSQL database over a relational database? That’s questionable, in this engineer’s opinion!

Ok well, I definitely need to handle a large number of dog rental requests. This idea is going to take off like crazy so I’m glad NoSQL is webscale.

The rest of this is a load of baloney, ChatGPT!

Who’s to say NoSQL will be faster at querying than PostGresSQL!? What if I need to get at my dog rentals by joining user information. What if I want to give free dog rentals on birthdays? That’s gonna be hard to do!

And NoSQL “can be more cost effective than relational databases” Phoey! Running my own datacenter can be more effective than using the cloud but I’m not gonna do that!

I also don’t think storing my dog rental data in unstructured form is a good thing. This is a mission critical app and I want the utmost data integrity.

Ok, let’s move on.

What about those Microservices? That seems like overkill to me.

That… Makes a lot of sense. Fair point on the pros and cons, ChatGPT. While I watched this answer type out I was full of judgment but then it ended on that note and I don’t know if I could have come up with a better answer myself.

Well done, ChatGPT, point for you.

Ok, so how about the deployment. Using “cloud services” is pretty vague. Let’s clear this up.

This is all suspiciously vague. This type of answer seems like it was written by a non-technical MBA student in their first year. Not even a mention of containers. Blasphemy!

Take that point away!

Ok, how about React Native. Sure, we’ll save some money and effort not having to code two different apps but the experience won’t be as good!

Hmmm… Ok, another good overview.

Point back, ChatGPT.

Finally, let’s ask ChatGPT about the CEO’s idea, after all, the best engineers know when NOT to build things, right?

Sounds like a winner to me! Off to build that app…

What did we learn?

AI is a good actor!

But seriously: despite seeing lots of posts/articles about how great ChatGPT is at various things, it still did quite a bit better than I thought it would with such a vague question about design/architecture. I was not expecting such well-rounded answers, even with caveats about weighing pros and cons.

It also does well with very specific information. You can see how diving into NoSQL gave many more specifics and even gave those nice caveats about when it might not be applicable.

The overly-confident tone it presents is off putting, though. For a new tool that is scraping internet articles for its answers, it sounds way more confident than it should be. And not all of the answers presented caveats about weighing pros and cons.

Using the tool is a great way to get started if you’re stuck or just learning. It does a great job of amalgamating information that might take you a while to collect with various internet searches.

However, I do hope that the confident tone can be scaled back before unknowing newbies use the tool to direct their software development efforts, whether it be for code or for architecture and design.

There is a great article I recently read called ChatGPT and the Enshittening of Knowledge that sums this up perhaps better than I can:

[…] in order to ensure that its answer fits what its model tells it is expected to come next, ChatGPT is also making shit up. So, we have a generic non-committal middle of the road representation of knowledge coupled with what, in a human, we’d class as “A-Grade Bullshitter” levels of self-confidence so they make shit up to support their argument.

The merit or trust rating of a book or journal or newspaper that cites the A-Grade Bullshit then increases the truthiness of the bullshit, resulting the next iteration of the question to the AI having EVEN MORE CONFIDENCE in their bullshit. […] This results in a classic data quality spiral where the incorrect data becomes accepted as fact and decisions or outputs that disagree are discounted. The Enshittening of Knowledge gathers momentum.

We’re still not even close to the eyes, ears and brain of a discriminating engineer.

You heard it here first: Learning the old fashioned way by trying and failing is still the way to go.

NOTE: The original version of this post appeared on DEPT® Agency’s Engineering Blog.

At my employer, we still use Heroku to host an internal project tracking application. Last year their Github integration broke and wasn’t fixed for almost two months, with very little communication from the company, that’s when I decided I would no longer recommend it or use it as an option for deployments.

A few months ago, a colleague enthusiastically recommended Fly.io to me. I trust his opinion a lot, and I gave it a shot and stood up a sample Node.js app on it. I was seriously impressed! An amazing CLI and I got an app with a database up and running in less than 30 minutes. AND I could do all the wonderful things I could usually do with Docker containers. Oh, and it still has a free option, unlike some others (looking at you, Heroku!)

I was so impressed, that when we had to host another internal app, I immediately reached for it. It’s worth noting that these types of Platform as a Service (PaaS) tools are not for every use case, but ours was perfect since it wasn’t a mission-critical app. That said, we still wanted some basic things for it like monitoring and being able to view logs, etc. Things that Heroku does splendidly.

To make a long story short: My experience was that though Fly.io has amazing promise, it doesn’t yet have the necessary features for operational use on a team. I’ll outline why in this article. It is great, however, for hobby projects, single developer applications. It’s especially great if you just need a (free!) place to start and plan on migrating to a more mature platform at some point soon.

This article may come off as critical, but useful software often has critics. I’m one of them. But, let me make it 100% clear: I am a Fly.io fan! I wouldn’t take the time to write this if I wasn’t excited by the promise it has.

Ok, on we go….

Why Fly is awesome

The whole platform is based on Docker containers. You can do everything from creation of projects to managing secrets via command line. This means everything can be scripted. It can use Heroku build packs. It can be as simple or as complicated (within reason, see below) as you want or need it to be.

Think of it as Kubernetes or AWS Elastic Container Service on easy mode. Though this ease of use brings with it a restriction the customization abilities of your deployments, and, as you’ll see as you read, some limitations.

Oh, and did I mention that it has a free option to get started?

Setting up an app

It’s very, very easy:

1. Download the CLI.
2. Authenticate with flyctl auth login
3. Run flyctl launch or flyctl apps create and point it to your Dockerfile or Docker image.

That’s it!

Plaintext Configs

Fly uses a fly.toml file that is used to define your application. All of your configs are here, at a glance and able to be version controlled.

You can also provide multiple definitions for different environments or variations as well using the --config flag on the CLI.

One thing to note is that there is no “override” option (a-la docker-compose), so you will have to copy/paste your definitions for now. No big deal.

Deploys / Free Builders

One of the nicest things of the developer experience is how easy it is to deploy your app for testing. You run flyctl deploy and it ships off your code to a free builder which builds your container image on their servers and uploads it to a secure private repo, then deploys it. It’s really awesome that they provide this for free, even as you scale beyond the initial free option. Other CI/CD offerings like Github actions cap you at a certain number of builder minutes a month then charge you afterwards.

Apps and Organizations

Fly has the concept of organizations, which are collections of individual apps. Individual apps are essentially Docker containers that can run and scale independently. This is also how you grant access to users to manage your applications.

Secrets Management / Environment Variables

Adding and managing secrets in Fly is very easy and a pleasure to use. You run a CLI add or update command and the variable is present to your app’s container automatically. That’s it. There’s even a very easy way to define non-sensitive environment variables in your fly.toml. This was probably the easiest and best way of managing environment vars I’ve used, bar none.

User Roles and Permissioning

There are NO user permissions beyond having access to an organization. You have organization access, you have the keys to the kingdom.

Additionally, your API tokens have access to everything you as an end user have access to. There is no way to narrow them down so they only have deploy access to one application. Most importantly, in order to hook up the command line utility to any deployment pipelines, you have to give your CI/CD tool full access to everything your user has access to 😵, including across organizations if the user is part of more than one. I would really like to see fine grained permissioning so that an access token only has deploy access to one application and nothing else.

To get around this, we created a separate “non production” organization where a whole development team can have permissions to freely configure and deploy the app. We locked our production organization down to a small subset of trusted committers/deployers.

Monitoring/Alerting

Fly gives you Prometheus metrics, displayed via a Grafana instance out of the box that are super useful. I really love that they used a well-known, open source tool to do this. It is a version of Grafana with features disabled, though.

Crucially, it does not have alerting enabled. I had to stand up my own instance of Grafana to be able to set up real-time alerts for errors on my app.

It was fairly easy for me to set up Grafana, and the Prometheus metrics from each app were available to plug into any Grafana instance.

I do like that the metrics are exposed and you can do what you want with it, but I had to sink about half a day into getting Grafana stood up with an SMTP provider.

Also, there is no way I’m aware of to export logs to a more digestible place like Splunk/Elasticsearch. You can look at logs via the web console or the command line only.

Databases

Fly does support running Postgres and other databases but they are really just running docker containers on volumes on fly itself. It doesn’t give you out-of-the-box High Availability or monitoring or easy snapshotting and restoring. To set up Postgres, you have to fork one of their repos.

Fly themselves claim that “Fly Postgres is not managed Postgres.” Managed means things like replication and automated restores (among other things). They even point you to Heroku for a fully managed version of Postgres. I commend them for being upfront about it, but it sure would be nice to be able to have a managed database alongside your app without having to worry about this. What they provide is certainly workable though!

Scheduled Tasks/Jobs

Surprisingly, there is no great way to run a Docker container as a scheduled task. There is a way to do it using the newer “Machines” API which is a bit buried in their documentation (search for schedule), but it only supports limited scheduling options and not cron expressions.

Do a quick Google search for Fly.io scheduled jobs, and you’ll see that many users have been rallying for this feature for a while.

Running Multiple Processes / Sidecar Containers

Related to the Jobs portion, but zooming out a little: Fly does not yet have support for sidecar containers, so running multiple processes requires adding a process manager into your Docker container. The addition of this feature might solve a lot of operational challenges we had!

If we could run a sidecar container, we could probably run our jobs or have a logging exporter to output logs to another service for us.

Security

One of the big draws of Fly is its ease of use on the command line. However, this comes at a cost - Because of the lack of fine-grained permissions noted above, it is incredibly easy to run a command to change the wrong application. And if your auth token is accidentally shared or exposed to a bad actor, they will have access to change a lot. Rotating keys also becomes problematic because you have to replace just about everything.

Speaking for myself, I would love to have an authentication configuration just for my non-productions applications to make sure I don’t inadvertently take down my production app.

Docker Limitations

Some things you take for granted in Docker may not be available, like for instance the ability to mount multiple volumes in a container. When I was setting up Grafana for alerting, I needed to mount two volumes to configure the app’s secrets securely, but only one volume was supported. Because of this, I had to work around it by copying files over and doing environment variable substitution.

Shifting Product Features

Fly is still in the early days, and they are adjusting the product quickly. This is great because you know that there’s a passionate group of developers working on this and making the product better. It’s not so great in that features are shifting and not being supported as well anymore. For instance, we created a Postgres database, and when we went to create a staging version of it, the option to create it was no longer available. When we went to the documentation page, we saw this:

When you dive into that, you find out that their new “Apps V2” architecture is supplanting their original Hashicorp Nomad architecture, which everything we built was based on. We stood up the app in June, and somewhere in the October/November timeframe, this announcement happened. Unfortunately, at the time of this writing (January 2023), the Machines API is just that: literally a REST API you have to send commands to. There is no support in their CLI for it or at least minimal support

In the course of writing this, I stumbled on a lot of information about how Fly’s Machines-based architecture will likely solve a LOT of these limitations.

One such example is that there is a Terraform provider for the machines API! That’s really exciting to see and hear.

In Summary: Onward and Upwards

I hope I’ve done a decent job of showing you the pros and cons of using Fly right now. Fly’s promise and advantage are that it is obviously run by a team of talented developers working to make a tool that they would want to use. They don’t have the baggage of a large company like Amazon or Salesforce holding them back. This is really exciting.

Very soon Fly will become a contender to fully replace Heroku for production application and will be much more simple and easy to use than something like Kubernetes or any of AWS’s offerings. Right now, it’s pretty much the perfect option for hobbyists or a team with one or two developers. When you need to get going fast and free, it can’t be beat. Just be aware of the limitations I’ve mentioned and how they may limit your ability to support your application.

In the meantime, if you need a simple-to-use deployment option for a production application on a team, my opinion is that you should stick with Heroku, or investigate using Elastic Beanstalk on AWS. Of course, there are other options as well (Aptible is a very interesting one I’ve heard about) but these are the ones my employer has experience with.

Note: I wrote the original version of this post for my employer, DEPT® Agency, you can view it here

I wanted to write in Markdown, too. Preferably using my preferred editor (VSCode) with my preferred Keybindings (Vim, thankyouverymuch!). There are plugins for Markdown in Wordpress I used, but they were pretty clunky too.

So, I finally bit the bullet and converted over to using Jekyll hosted for free on Github pages. You can see the source code for the blog here.

Jekyll has so many wonderful theming options to choose from for a backend engineer who has no design taste! I chose the Clean Blog Jekyll theme, which looks a lot like the old theme of my site, but a little more fresh with nicer typography.

The process of the export was very easy with help from the Jekyll Exporter Wordpress plugin and Bob Gale’s “From Wordpress to Jekyll” post. I kept all my canonical blog URL’s with barely any effort.

What a wonderful change: I’m writing this with my Vim keybindings, and I can source control the whole thing. I’m in heaven here, folks.

And you, dear reader, benefit from a crisp new design and near instantaneous loading times because the whole site is static. No more waiting for Wordpress to load folks, you just get to my content faster. Which I’m sure you’ve been dying for.

Happy new year!

It all started with a simple request to see how hard it would be to setup a Mastodon server for my company.

Sure, easy. I’ll do it!

So began my journey into creating a Mastodon server.

But, before we get into all that …

Why would I create my own Mastodon server? Isn’t that why Twitter exists?

It would be hard to avoid hearing about Mastodon lately with all that’s happening around Elon Musk’s takeover of Twitter: Journalists being banned, accounts in violation of policies for links to other services, and constant unpredictability with policies being changed daily.

But, even though you’ve heard of Mastodon, you may not know how you use it - here’s a good resource to learn a little more. Also, if you’re not already signed up: You have to find and choose a Mastodon server to use, a key difference from the centralized approach of Twitter. To get you going faster, I’d recommend creating an account on Mastodon.social (one of the biggest servers). Then, once you get that going, use Movetodon.org to import your Twitter following list into Mastodon.

But why on earth would you want to create your own Mastodon server? I can’t state it any better than Julien Deswaef in his article Your organization should run its own Mastodon server:

… just as with email, you can send a message to anyone in your organization, and at the same time have a conversation with anyone outside of your organization. All this is transparent and your correspondent’s email address lets you know that you are actually talking to someone as part of a particular organization.

By using your own domain name, your brand, you are creating a recognizable social presence in the Fediverse, without the need to associate it with facebook.com, twitter.com or instagram.com. No need to worry about someone else squatting your name either. Your domain name is what will get people to trust that the Mastodon accounts under it are legitimate and official.

And once you run your own corner of a social network, you can decide who you invite there, what are your rules of engagement and code of conduct. At Thoughtworks, we’ve allowed all our employees to open an account on our Mastodon server as this is aligned with our culture and our practice of being quite vocal and transparent about what we are passionate about.

Having some piece of mind for your company’s brand to exist on a social network that you control fully is huge.

Ok, now that we’ve gotten that out of the way…

The Context

Jumping ahead a bit, one thing that I couldn’t really find was a nice, introductory article explaining the architecture of Mastodon. I found myself struggling to set up infrastructure I had no context in. So here’s a crash course that will help you understand things a bit better before diving in.

Mastodon is comprised of just a few main components, all within the same open-source codebase:

(Diagram Source)

The main web application. This is a Ruby on Rails application that runs on Puma. This includes web pages themselves as well as APIs.

Background Job Processing. This runs using Sidekiq, a Ruby job processing system. These jobs handle things like handling requests from other Mastodon servers (known as federated servers) for messages and follow requests, handling notifications, and other things.

The streaming API. This is a Node.js application that has Websocket API’s for real-time updates for the various parts of Mastodon’s data model. Think of things like getting notifications in the app for new follows.

Another important thing to know is that Mastodon created and implements the ActivityPub spec. This is a specification managed by the W3C for decentralized social networking. I won’t go into the details about ActivityPub here, but it’s super important to know about the concepts in it to help you with setup, more on that later.

How Federated Servers Communicate

I think perhaps the most confusing thing is understanding the flow between different Mastodon servers. So, let’s take the example of Following someone on a different server:

1. A user clicks “Follow” for someone on a different Mastodon server.
2. The Puma web application will take the federation information and lookup how to contact the other server and send a follow request to that server. The endpoint that’s called on the other end is something like POST /users/{followed_remote_username}/inbox. This puts a message in the remote user’s “inbox that they have a follow request”.
3. On the remote end, the API endpoint puts a message in the Sidekiq ingress queue. When the job processor picks up that task, if the user doesn’t need to approve follow, the server will respond with a corresponding call back to the original users inbox that the follow request is accepted.

This is all part of the ActivityPub spec but is kind of hard to follow, and if I’m being honest, I’m still not sure I got it completely right (comments welcome!).

The takeaway is that background jobs are really important in Mastodon, and so is the ability to securely connect between servers through HTTPS connections. This process is fraught with SSL errors and layers you’ll need to debug if the connections don’t just work.

The Infrastructure Needed

To host this beast (ha ha) here are a few things that need to be set up, and a couple optional things:

An online web application container. Something that can run Ruby processes. Could be Docker, could be a Virtual Machine. Pick your poison.

A PostGresSQL database. This is the heart of your Mastodon instance storing user accounts, messages, etc.

A Redis data store. This is used for the job queueing system described above as well as caching.

A static file store. This is used to store uploaded photos/files for posts. Think AWS S3, or similar.

A reverse proxy or load balancer. You really don’t want your web application directly hosting requests. You’ll want some way to throttle and distribute those requests and also put in some security checks as well. You could use something like NGinx for a reverse proxy, or in our case, we used an AWS Application Load Balancer.

Optional: An ElasticSearch cluster. This is used for advanced searching capabilities. I didn’t set this up so it’s not included in this write-up. If I end up setting one up I’ll update the post.

Choosing a path

There are affordable hosted Mastodon server offerings, but really part of this exercise was learning more about the architecture of Mastodon and what goes into setting up a server. After all, we’re engineers. We can do it better ourselves, right? 🫠 Kidding…

If you have the knowledge, it makes a lot of sense to stand up Mastodon yourself on cloud infrastructure instead of relying on a hosting provider: You can be assured you’re in control of your data and treat Mastodon just like any other app you develop. It’s also advantageous to be able to really tweak your cloud resources to your number of users to save money or scale up. whereas hosting providers charge for the price of Mastohost is $89 a month for approximately 2000 users, and the price for toot.io’s 1000 user plan is $89 a month. Also, as of this writing, Mastohost isn’t even accepting new customers according to their page. That’s kinda important especially with the recent influx of Mastodon users.

As for cost, I’m anticipating it will be a wash compared with hosting providers. The AWS estimate for our first month’s bill is about $130 but that includes lots of experimentation and also a NAT Gateway, which can be avoided if you’d like to be slightly less secure (exposing your containers publicly, which is generally a no-no). See my update at the end of the article. It may be more cost-effective to use one of these services unless you need to tightly control your infrastructure!

I’m most familiar with AWS and a . big fan of containerization, so I thought I might go in that direction. It helps that Mastodon has official Docker Images! Finally, I knew if I was going to go with AWS, I wanted to use an Infrastructure as Code (IaC) tool so that I could share the setup with the world, and you, dear reader. I’m most familiar with Terraform so I chose that.

With all my preferences, I set out to see what already existed to help me out. One of the huge benefits of using IaC tools is being able to leverage existing open-source code to do this type of thing. After some false starts, I landed on a project called mastodon-terraform by r7kamura. It was a beautifully laid out Terraform project with modules and a great setup for providing configuration variables. It also used Elastic Container Service which is a “just complicated enough” container orchestration engine.

The problem was that it was out of date, having been touched last six years ago. Mastodon had only released its first stable release in 2017, so I assumed much had changed between then and now. It also did not use Fargate Managed ECS Clusters, but instead EC2 servers which can be cumbersome to maintain, even with Terraform.

That was as good a place as any to start!

The Takeaways

In the course of a couple weeks of off and on work, maybe about 20 hours total, I got a server up and running, ready to scale out should our community take off. My company shared the forked repository for all to use here: deptagency/mastodon-terraform-aws-ecs (MIT Licensed).

I hit a couple bumps in the road making changes to container definitions in ECS to be in line with updates to Mastodon since 2017, but other than that r7kamura’s work was a fantastic base and I thank them greatly.

I also hit a knowledge wall at one point where messages from other servers were not being received by ours. I spent a lot of time with networking settings, but in the end, posted a question on the Mastodon Github discussion forum which was super helpful and pointed me in the right direction.

My biggest challenge was finding high-level information about mastodon’s architecture and infrastructure. Mastodon has all kinds of fantastic documentation about the details of how to set up a server and how to configure it. They also have great introductory material on how to use and administer Mastodon. Everything in between is a bit of a black box. I hope that this post helps someone else fill in those gaps.

Update 2/2023: The cost for the past couple of months ended up being about $170 a month! The majority of that was (surprisingly) the NAT Gateway to keep the containers isolated from the public internet. This could be avoided by allowing containers to be accessed through the internet. We also ended up dialing back the RDS instance size to save a bit.

I wrote the original version of this post for my employer, DEPT® Agency, you can view it here

I learned a lot of lessons about what to do and what not to do during this process.

Why Microservices?

This particular application team wanted to rearchitect with Microservices but didn’t seem to have scale problems. They had a modestly large user base, but nothing that was really stretching the scaling boundaries of their current app.  We found this peculiar.  What was their goal in using Microservices, then?  At this point, everyone likes Microservices, but most are cognizant of the overhead they can incur as well.

It took me and the team a bit of time to figure out that they wanted Microservices to accelerate the pace of development. And they didn’t just want them, they wanted them NOW! Ah, this was a different beast that required a different weapon to take down.

Knowing their primary goal was time to market, we worked towards trying to scope new services with proper domain boundaries rather than simply just creating a new service for each portion of the domain.  Our challenge was to keep things as simple as possible in a Microservices architecture.

Mission impossible? Probably. 😃

API first development

The desire to quickly deliver a new feature set with this architecture forced our teams to work within an API contract driven development model.  This allowed some developers to run ahead on frontend development by agreeing on what the backend REST API would look like while we built it out.

We documented the API’s with OpenAPI specs from the very beginning.  This was crucial.  Our frontend team used these specs religiously.  Later, we were able to take the open API specs and use tools like express-openapi-validator to do automatic validation and swagger-to-ts to generate Typescript interfaces from this (if you haven’t gathered by now, a bunch of our services were in Node.js and Typescript).  Other teams were using Java, and because OpenAPI is, well… open, the tooling for other languages exists as well and can easily be supported.  Our initial investment in API driven development paid off in spades by avoiding writing a lot of boilerplate code.  This all worked brilliantly and we would repeat it again in a heartbeat.

Side note about API’s:

At the beginning of the project, we had a lot of debate about whether to use REST API’s or GraphQL API’s.  One thing that jumped out at us was that something like GraphQL didn’t need OpenAPI, it was built in!  However, we ended up going with RESTful API’s because we were running lean - We needed to get going fast.  We made the decision that the time necessary to setup GraphQL resolvers and other infrastructure was best spent elsewhere.  I’ll save the debate about whether or not we were right for another post. 😜

Service boundaries are kinda important

Since scaling wasn’t our primary goal with this architecture, we set our sights on defining the right service boundaries to choose. Our first feature handled a somewhat broad swath of functionality, so we created a service with a larger set of capabilities than you’d typically see in a microservice.  This was instead of trying to carve off a small niche of functionality to allow it to scale independently.  We had plenty of groundwork to lay, so it was nice that we could start with just one new service and still complete a large feature set.

However, this time to market focused approach of determining service boundaries led to bad decisions (big surprise, huh?).  Because we were under the gun to deliver, we didn’t get a chance to fully understand the business domain before we dove in and created a new service.  This led to us (honestly) confusing where various API’s should be located:  Should they be in the new service we were creating?  Should we have created multiple services?  Should they be in the legacy application temporarily?  Should they be a completely separate service?

In hindsight:  We should have done more experimentation with service boundaries using real working code, and refactored as we went if we needed narrower service boundaries.

Using your cloud provider to the max

One major factor of the success of our project was staying within one cloud provider’s ecosystem.  In our case it was Amazon Web Services (AWS).  This same principle applies for any of the major cloud providers though: Using their utilities whenever we could saved us time and kept things smooth and moving quickly.

Our domain was hosted with Cloudfront.  We used Application Load Balancers (ALB’s) to balance traffic.  We used S3 to host our SPA’s static assets.

On the API side, we deployed our services into Docker containers using Elastic Container Service (ECS) which stored data in databases managed by Relational Data Service (RDS).  It worked splendidly.  The one place where we strayed from pure AWS was that we configured things with Terraform instead of Cloud Formation.

For running containers, we toyed with the idea of using Kubernetes, but decided that seemed like using a jackhammer when only a hammer was needed.  ECS is “just complex enough” for an orchestration engine, in our experience. A year out, that still seems like the right choice.

Pay attention to the glue

One of the executives described moving off the old system while keeping it running as: “Changing the tires on a bus while it’s moving.”  Which is an apt metaphor. So, how were we going to share data between the systems and gracefully move off over time?

One thing that we did right was picking an API gateway that was endlessly customizable in one central place.  Because we were working in (mostly) a Node.js stack, we chose express-gateway as our entrypoint to our Microservices.

In the year at this client, this has been the biggest question:  “If you’re so bought into AWS, why not just use AWS API Gateway?”  There’s a few reasons:

First, we were excited by the fact that express-gateway was basically just glorified Express.js middleware with a pre-existing set of routing policies built in. It was open source, backed by the Linux Foundation and recommended by Auth0.  Using it, we could get the best of express along with not having to code our own proxying or filtering logic.  We were able to use existing policies like JWT authentication while also writing our own policies for those strange, unforeseen situations that inevitably come up during a major technical transition like this.

Second, we knew that authentication between the two systems was going to be challenging.  The old system used session based cookie authentication, and our new Single Page App needed token based authentication.  How were we going to bridge that gap while not propagating this issue to each of the underlying services?  We could customize the heck out of AWS API Gateway using Lambdas, but we decided that having a central point where we could keep this code and run pipelines was preferable.  We were able to use express-gateway custom policies to bridge this gap in authentication nicely, all in one place in code, without having a bunch of Lambda code sprinkled around that was hard to track.

Third, we knew that this transition state between an old and new architecture was temporary.  We constantly told everyone that if we did things right in this “semi customized” API Gateway, that someday, when the old system was completely retired, we could completely get rid of it and move to an off the shelf solution like API Gateway. This did end up happening eventually after I left.

Don’t kid yourself:  The old Monolith isn’t going anywhere soon

The tech team wants a new system, and the product team is smart enough to know (and so are you) that “The Big Rewrite” approach is a recipe for disaster! You want to be lean and move your old system into the new world gradually, delivering value along the way.

But, you can’t pretend like your old system is going anywhere soon, even if you really really want it to.  We spent a lot of time setting expectations around this with the client.  In our case, and I’m sure many others, the old system will still be the primary system the business runs on for a good while.  If the old system is shaky enough, it will take down the new system as well.

We spent a good amount of time shoring up the older system, which we had to work hard to justify.  And that still wasn’t enough. Moving an application to Microservices doesn’t have to be all or nothing.  You can take steps in your existing Monolith to get it ready to be split up into Microservices.  It’s important that your product team understands this for roadmap planning.

Don’t short change your investment in the foundation

A wise colleague once told me:

You can’t fill a garbage truck with cement if you want to lay a foundation.

Along those lines, our primary learning point in this project was around foundational Microservices architecture: You cannot build an “MVP” of Microservices infrastructure.  If you have even the faintest whiff there will be more than one service, you have to go all in and invest in the necessary infrastructure to support many services.

Because of the desire to deliver quickly, our Continuous Integration / Continuous Deployment (CI/CD) pipelines were created using an MVP approach.  This was a mistake.  The minute it became time to create services number 2 and 3, we felt the pain of the missing capabilities we didn’t add.  An example was sharing build artifacts across environments.  These missing capabilities caused each service to take a long time to build, and was repeated for each environment to save a little time at the beginning.  After a couple of services, this was a nightmare.

We didn’t have time for things like standards on API formats from the get go, and instead learned in service 2 or 3 that we needed them.

The time spent up front building this foundation would have paid off in spades.

Wrapping up

• Microservices are serious business.  You can’t shortchange the time and effort needed to do them right. of where the product is going and.
• Use your cloud provider as leverage to move faster
• Have a deep understanding of the business domain and the roadmap going forward in order to architect service boundaries properly

And of course….

Happy microservicing!

Note: I wrote the original version of this post for my employer, DEPT® Agency, you can view it here

“How can I move faster?”

I’m sure we’ve all heard this many times from bosses and clients. Regardless of team size, industry or maturity, I consistently see teams struggle to scale. Even the ones that do it well have some serious learning to do. In some extreme scenarios I’ve heard “I am spending more on software, yet, somehow, I’m seeing less software get released. How the hell is this happening?”

Those of us who’ve been in software for a while are probably very familiar with Brooks’ Law and The Mythical Man Month which states:

Adding manpower to a late software project makes it later.

This is what you’re seeing when the situation described above occurs. It makes it even worse when your team isn’t set up to scale.

Why should you care about scaling your team? Here are some symptoms of a team that is struggling with growth:

• Problems organizing work - Entire teams stuck waiting on product requirements on other teams, wasting time.
• Too many hurdles make the slightest of changes to production take a very long time. This leads to very frustrated business partners.
• Paralysis by analysis - Teams stuck debating on how to implement something, again, wasting time.

In this post, I’ll start with the less technical ways of addressing these challenges and end with the more technical ones.

First up, the people!

The Talent

This will sound cliche, but it all starts with hiring. You want to know how to scale your team before it’s too late . Here are some things to consider when hiring your software team.

Start with strong technical leadership. Hire this talent first. By “technical leadership” I don’t mean “managers.” I mean player/coach talent who do the work while building out a team , then move to full time leadership.

When picking your technical leads, it’s best to prioritize communication skill and curiosity over technical ability. You want a technically sound leader, but more importantly, you want a leader who asks good questions, exposes shortcomings, and knows when they are in unfamiliar territory. This type of person is also someone who can communicate with you and about your business with ease. Someone who can talk tech and also business is rare, so when you find it the cost may be high but it will be worth it.

If you nail your tech lead(s), you’ll have a person who can ask the right questions to get projects started, who knows how to staff them with the right balance of people, and who will also contribute. To vet this person, leverage your network of technical people to help. Depending on your plans, you may want to hire a few of these people.

After you hire your technical leaders, you want to make sure you have a good balance between leaders and individual contributors. Trust that your tech leads will lead this hiring effort and let them know how much you want to be involved. Don’t just hire all senior engineers, or you’ll have a situation where you have stalemates on technical decisions all the time as Senior Engineers tend to have strong opinions.

Ideally, you have a balance of people who will want to lead the charge, and people who are more junior and eager to learn. In all of these roles, curiosity is a key trait to look for. Make sure you hire Senior Engineers who want to mentor and check in with them frequently to make sure their mentoring duties don’t interfere with their daily work too much. You’ll know you’re on the verge of having too many junior developers when your senior engineers say mentoring is taking up too much time.

Finally, make sure you have a large enough product and/or business team to back up your technical team to help drive work forward. A technical team without enough product owners and/or designers is as good as not having a software team. Of course you want independent teams who can drive work, but if you don’t have the business expertise at your disposal you’ll be unpleasantly surprised when your app or platform looks like it was built by engineers.

Organization

After people, the next most important thing is organization - Of people and work.

The big theme here is: Less is more. Put some basic constraints in place and trust your teams to do the right things. You don’t want everyone going wild, so things like making sure everyone’s using the same language might be a good constraint. But, determining how individual teams run meetings and processes is only going to create frustration. Again, it’s amazing what an empowered software team can do if you give them goals and leave them to their own devices.

As a wise man once said:

If you want to build a ship, don’t drum up the mento gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.

• Antoine de Saint-Exupéry

Team Structure

You don’t have to nail the team structure from the get go. Just keep in mind that you’ll have to create more teams and have a list of “things to figure out.” Make sure you’re transparent about what has and hasn’t been figured out. Figuring things out as you go and being honest about it is a lot better than trying to anticipate everything at the outset. You’ll save everyone’s sanity and time. Share the plans for growth with your teams and ask for their input. Software teams can be extremely proficient at helping with these things. They can let you know what they’ve seen work and more importantly, you want them bought in since they’ll be living with it every day.

Inter Team Communication

Figure out a way to communicate across teams. Sometimes this is Slack channels, sometimes this is a Scrum of Scrums with your technical leads.Whatever it is, just get a basic cross-team communication framework in place and let the teams adjust it and run with it as they wish.

Organizing Work

Finally, I should talk about organization of work Strive to have about a month’s worth of work shovel ready for all of your teams. I use the term shovel ready to describe work that is well-defined enough for teams to go and execute on without being blocked. This is an art, not a science; if you over define the work, you risk alienating your team. However, if you don’t define it enough, you’ll have a bunch of developers looking at each other wondering what you meant by “I want the software to be easy to use.”

To get to this point, make sure that your technical leaders and product leaders are working together regularly to help strike the right balance and keep defining the work to keep it ahead of the individual contributors on your teams.

When you have a team who has enough definition and enough freedom to ask questions and try out different solutions when implementing, you’ll have a team of engaged, productive people and you’ll be pleased with the results. It’s also extremely important to have a team that feels collaborative and is part of decision-making when things change.

If a feature or change isn’t working out, the team shouldn’t feel like it’s a reflection of poor work, but rather that they are working together to make a better product.

Communication

In the previous section, I discussed what can go wrong when a software team isn’t built to scale properly and some strategies around hiring and organization that can help mitigate some of those problems.

Next, I’ll tackle another area that’s ripe for problems relating to scaling teams:

Talking and Stuff. Aka - Communication.

The Blessing and Curse of “Tribal Knowledge”

Tribal Knowledge is an often-used (and very unwoke) term for the stuff that your team knows but isn’t codified yet. When you’re starting up, there’s no time or need to write those things down or automate them. It’ll just slow you down. The fact that this knowledge is just known on the team makes it a well oiled machine.

But as you grow and new people come onto your software team, they only find out about these things by happenstance or even worse when an issue comes up and they’re not sure how to fix it. You could consider the process of gaining this knowledge a tax on every new person on your team. To become fully productive, they need to absorb all that knowledge and the only way to do it is to give it time.

This knowledge can be simple things like where the link to the deployment system is and what branch production is deployed from. Or, it can be skeletons (as in skeletons in your closet). I often find that companies that grow quickly also have skeletons in their closets that only a handful of “the old guard” know.

In the worst case scenario, the old guard views having this knowledge as a badge of honor and job security and holds onto this knowledge. In the best case scenario they have these issues documented and potential tickets documented knowing that they need to be fixed.

Regardless of where you might fall, it’s best to get ahead of this as your team is starting to grow.

Make sure that you reward transparency and information sharing, not information hoarding. The default reaction to a team member saving the day during a production incident or fixing a bug should be to ask “How do I make sure that doesn’t happen again? Is what you did written down somewhere? Could other developers do that if I needed them to?”

Make sure your technical leads set goals for their teams to codify their knowledge. Simply dumping all the knowledge into a document or wiki is not enough, this needs to be thoughtful and strategic.

Here are some other ideas to consider to get ahead of this as you think about scaling your teams:

• Set aside some time periodically to prioritize automation of tasks that require tribal knowledge. There’s nothing better than having workingcode instead of documentation.
• Make sure that code comments which document the “Why? Rather than the How?” are prioritized and vetted during code review processes. As the linked article explains, the code is the how, and what’s more important is the why. This is especially important when something weird is happening. Keeping this documentation as close as possible to the code itself is much better than distributing it in different places like a document or wiki.
• Have a [Runbook](This knowledge can be simple things like where the link to the deployment system is and what branch production is deployed from. Or, it can be skeletons ☠️ (as in skeletons in your closet) . We often find that companies that grow quickly also have skeletons in their closets that only a handful of “the old guard” know.

Curiosity

In our experience, the best developers on the best teams are very, very curious. They don’t just want to build the thing you ask, they want to know why what they’re building will be valuable. The best developers and teams know that if they understand the core of what a business is after, they can lend the strengths of their technical discipline to making products better.

Don’t treat a developer with lots of questions as a nuisance; embrace them and foster a culture where asking those questions is not only rewarded but expected. If your software teams aren’t asking lots of questions, consider that a potential red flag. Perhaps your tech leads are hoarding information!

Organize around information sharing

Make sure that between you, your technical leads and individual team members there is a clear sense of who makes what decisions, who needs to be informed of them, and who doesn’t care at all. There’s a tool for that! RACI matrix anyone? You don’t have to use something as formal as a RACI matrix, but be inspired by the idea. Strive to make sure your team can make as many decisions as possible independently.

When a team knows they’re empowered to make a decision, they will move much faster. And when they can’t make a decision, if they know exactly who to go to, then that decision will get to the right person a lot quicker. This, of course, assumes that when a decision needs to be made outside the team, that decision-maker makes it a priority to get that decision to them fast. Reward unblocking your teams throughout the organization.

Establishing a RACI matrix (or something like it) before your teams scale and updating it as you go is very low effort and will pay off in spades.

The Technical Things

Getting Started

Developers often have a difficult time getting an application running on their local machines. In the worst cases, it can take a developer more than a week to get the full system running so they’re ready to start developing.

Making sure that other developers can easily get started running a system locally is something that’s often overlooked early in a project. As your team grows and new people have to create this setup (or, existing people get new computers!) this slows you down immensely. Combine this with the amount of “Tribal Knowledge” that a new person needs to be productive and you can see how this will slow down the productivity of your team and their ability to scale.

So how can you get ahead of it? Prioritize automation. Whether it be through specific talent like DevOps or Site Reliability Engineers or Release Managers, or just making it one developer’s priority to script things - Make sure that a new developer can get started in a few hours at the very most. At the very best, one script does all thesetup for a new developer - they run the script, go have a coffee, and come back to a workstation ready to develop on.

Be mindful that different developers have different preferred platforms they use. Some prefer to use Mac, some prefer to use Windows, others Linux. I are not fans of dictating tooling, so prefer automation methods that work across platforms like using Docker images for development setups, or provide different scripts for different platforms.

Oftentimes, the effort you put into scripting a local setup can be directly translated into deploying the application to a real environment. Even if the scripts aren’t directly used, you can usually translate them into a new language/framework because everything you need to set up is in code and not just written down.

Which brings us to our next point …

Make significant investments in Automation/DevOps

One surefire way to kill the ability to scale your team and software is a lack of investment in automation and “DevOps” (side note: I’m using that word even though I hate what it’s become!). I have seen many projects where it’s time intensive and risky to get software into production because processes are manual. In the worst cases, nobody fully understands how to get the software into production at all.

In some of these same projects, there is little ongoing investment in their DevOps platforms to solve the problem. Instead, more resources go to the development team to crank out more software that is risky to be deployed.

Even in the earliest phases of your project, make sure you have a fully automated Continuous Integration / Continuous Delivery (CI/CD) pipeline. Regardless of the solution, it should be as easy as pushing code to a branch or clicking a button to deploy to production. Whatever you do, don’t give into the temptation to manually push code out! It will come back to haunt you before you even get the software out to users. The investment in a CI/CD pipeline will pay off even before production releases because it will allowyou to get test versions of your software out faster to be QA’d. These days with so many easy to use options for CI/CD pipelines, nobody has a good excuse for not having a good CI/CD pipeline.

In concert with CI/CD pipelines, invest in Infrastructure as Code (IaC). I recommend Terraform. These tools let you describe your cloud environment and create it with one command. Using these types of tools not reduces your risk of environments being different, and also gives you the ability to spin up new environments very easily. New environments should ideally be button click. This will allow you to spin up a new testing environment for a new feature set, then easily tear it down, saving on cloud compute costs.

Make sure your environments mirror each other as closely as possible. It takes discipline to make sure that this stays in place. Make investments in anonymizing production data to copy to testing environments to make sure features work with production-like data.

Testing

This one isn’t groundbreaking, but needs to be said: make investments in test automation and integrate tests into your CI/CD pipeline. Do this as early as you possibly can. It’s much harder to add tests to an existing application than to adjust them as you go.

I often hear from clients that they invest heavily in automated testing. Yet, when it comes down to it, their strategy is just not working and releases of software still require extensive manual testing.

It’s easier to start by saying what doesn’t work in this area:

• Relying solely on unit tests in each component of the application. This often just tests one particular case, but functionality of interdependent features often gets missed here. This will inevitably lead to more manual testing.
• Having too many automated front end tests, such as in Selenium or Cypress. No matter how hard you try, these tests will be brittle and will break over time and require a lot of effort to maintain. What’s worse than no tests at all? Tests that break all the time and you can’t rely on them. Rely on these types of tests for absolutely crucial pieces of your software, main use cases and smoke testing to make sure that an app comes up after a restart. Do not try to test everything with them. I’ve yet to see an organization succeed at this over time.

What does work is automating API tests. It doesn’t matter what you use, but being able to work through key functionality cases using just a series of API calls is a good way of automating tests that strikes a good balance betIen making sure your whole application works together and being easier (note not easy) to automate.

Having API level tests that go all the way from login, to storing data in a data database, to logout allows you to test a very wide surface of your application with even one test case. It can be difficult to catch every corner case with these types of tests though, so you have to still rely on unit tests for those situations. A good mix of API tests with unit tests for individual components is what you should strive for.

The challenge with automating API tests is having the necessary infrastructure in place to allow for full application testing in an environment that can be torn down and built up at will. Quick throwback to having CI/CD pipelines and IaC tools. If you have a good CI/CD pipeline in place, this should be easier. You can do this with quicklydeployed databases using Docker containers so you don’t even need to set up a whole cloud environment to do these tests, which will make them quicker. The sticky wicket in these tests are third party API’s which you don’t have control over. Use a network mocking tool like Mock ServiceWorker for Javascript to make your tests as seamless as possible in this regard.

I also suggest having code coverage targets only for areas of the code with high cyclomatic complexity. Arbitrary code coverage targets will ensure your useless methods get covered as much as your most important ones do. Focus on the complicated ones and let API tests cover the rest.

Code Organization & Decoupling

One of the most significant (and obvious) pain points on large software teams is when multiple people are doing work on the same piece of code at the same time. The team ends up spending more time fixing source control conflicts and coordinating than actually producing value. There’s no perfect way to organize code, but if you give it some effort up front and adjust as you go, it’s a pretty easy discipline to get into, especially if it’s part of your code review process.

Make sure from the get go that you’re not putting all your code in a few modules or files (this one’s pretty obvious, but worth saying to reiterate). Once you hit a certain point, separate files will become unsustainable. You will smell the code rot from a mile away.

One other idea to consider is the idea of contract driven services (lowercase s) within the same application. Many of the benefits of a Microservice architecture are derived from the concept of Domain Driven Design, specifically the idea of a Bounded Context. This is a fancy term for drawing smart boundaries around portions of the system, creating contracts for interacting with them and restricting the communication between boundaries to use that contract.

If you can come up with the right bounded contexts for services within your application, you can then work to define contracts for those contexts using simple code documentation or even going so far as to use things like OpenAPI. Yes, using OpenAPI even if you’re not making a separate deployed API yet.

The simple act of being more explicit about the contract of a set of services is the important part. By defining the contract of a bounded context, you can then split work out across teams and make sure that they work with each other’s code through those contracts.

Once you hit a certain point, separate files will become unsustainable. You will smell the code rot from a mile away. Once you start to sense this oncoming disaster, it probably makes sense to create a separate API or application for some portion of your system. Then, if you’ve created the right bounded context and have a well defined contract, this exercise will be a lot easier, and there will likely be a team who is already well versed in that particular service that needs to be split out.

Speaking of services…

What about Microservices?

The past ten years or so the buzzword “Microservices” thrown around as a panacea to fix many problems in scaling and organizing software. And of course, as with many technical buzzwords, there’s no one definition of what a Microservice architecture is. The most important thing to remember is: Don’t conflate a well organized application with Microservices. A well organized application with separate domains is always a good idea. Microservices may or may not be a good idea.

There are many situations where a microservice architecture will help, but to “Do Microservices Right” you need all kinds of foundational architecture in place. For example, have you addressed these questions?

• How are you going to handle deployments across dozens or perhaps hundreds of services?
• How will you handle transactions betIen services?
• How will you handle discoverability of services?

Those are just a few of them. If it’s not obvious, you can quickly get in over your head, especially if you haven’t done it before. Even the godfather of Microservices himself says “You must be this tall to use microservices.”

Start with simpler options like those described above in the “Code Organization” section. It probably does make sense at a certain point to have a system with multiple services or applications. But do you need to do it for every little piece of your application? Probably not.

When you use the ideas I outlined in the Code Organization section, you’re effectively forcing your teams to work as though they were using a Microservice but without all the headaches of separate deployments, dealing with cross service transactions, etc. You’re getting yourself more than 50% of the way there, but keeping things simpler, and more importantly, letting your teams scale out more.

Parting Thoughts

You can probably tell by the sheer amount of content given to non-technical topics here, I feel strongly that an organization that can scale is driven by people, culture and communication, more so than technology. Remember that languages and platforms are tools to use to get to a goal, not the goal itself.

Good luck growing your team!

Note: Me and other coworkers wrote a different version of this post for my employer, DEPT® Agency, you can view it here

About 8 years ago, I started the biggest journey of my life: Having kids. Before my wife and I had our first child, to say I was anxious is an understatement. Of course I was happy too, but a lot of the time my thoughts would clash between things like "Will she be healthy and have two arms and legs?" and "How will I function with no sleep!?". It was a constant tug of war between selfless and selfish thoughts, causing spirals of anxiety for me.

As a software geek, I kept thinking about the amazing article Jeff Atwood wrote called "On Parenthood" where he presents one of the best data visualizations I've ever seen related to the raising of children.



After our daughter arrived and we spent our maternity/paternity leave groggy and caring for a beautiful new infant, we settled into the rhythm of having a newborn. We got used to losing sleep and started making sacrifices for ourselves for this new bundle of joy. We did this without a second thought. It just happens, and when you're in it, you're in it. Looking back on it, the anticipation of having a newborn was actually much worse than having a newborn!

But, after a few months, something happened that I didn't expect. Everything else got easier.

Drama with friends? Whatever, I don't have time for that, I've got a baby to take care of! Drama at work? Ergh, I'll keep my head above the fray - I need to focus on work so I can get home and take care of the baby! Should I use NoSQL or relational databases at work?

It's that last one I want to talk about a little more. This is the type of decision I would pontificate endlessly over. I'm a consensus-seeker too, so my own internal gnashing of teeth would be amplified by asking for others' opinions.

But after my first was born, the answer to these types of questions came a lot more quickly. In the case of SQL vs NoSQL, for instance: "Well, this seems like a wash, so I'll just pick the one I'm familiar with and get the job done, so I can spend more time with my family."

This decision-making process could sound like a bad thing - Like i'm cutting corners because I want to spend time with my family. Honorable on a personal level, but negligent professionally. Maybe, but I know that I would never be negligent about making a decision, it's just not in my nature. But knowing that a beautiful living being was waiting for me at home really forced me to (subconsciously) make a decision a lot quicker.

It got even more crisp when my son was born four years later!

With some hindsight: This type of pressure is the best thing that has happened to me both personally and professionally.

It has forced me to become a much more crisp and quick decision-maker. It forces me to prioritize getting things done over making things perfect. It has also made me more scrupulous about when attention to detail and striving for perfection is important, and when it's not. And finally, it has forced me to become well versed in Essentialism and the subtle art of saying "No" to things. It's amazing how much better your life becomes when you become comfortable with saying no.

As a Software Development manager, all of this also has positive effects for my teams as well - I'm much more comfortable delegating decisions to them. Being judicious with my time forces me to let those who are closest to the problem at hand make the decisions. Of course, I always want to be informed, but I make it clear that my input shouldn't hold them up.

I have found that often in the world of software, shipped is better than perfect. Feedback from users is better than spending time and money making something that may not be used by anyone. My kids helped me realize that and I thank them for it.

:drumroll: Announcing…..

The JVM Jargon Generator

Years of toiling away with Java Virtual Machine (JVM) technologies made me realize that the only thing that really matters when developing in the JVM space is that you can intelligently talk about it. These complex projects rarely get deployed to production or used by users, so it’s usually sufficient to just have a headline and be able to talk circles around your businesspeople to confuse them. Then, you can go have a beer.

Written in state of the art Kotlin using complex AI (if/else blocks and switch statements) it is truly a groundbreaking piece of technology.

Please send me copious amounts of money so I can buy a beach house and retire. Thanks!

Postscript January,2023: Since Heroku decided to shut down their free tier 😡, I moved this app over to Fly.io which has been an absolute pleasure and something I plan on writing about.