Which Generator builds Markdown the fastest?
Blogging. The quintessential starter project for most—if not all—site generators. Here’s a few samples:
- Next.js guides new developers to
Learn Next.js“by creating a very simple blog app.” - Gatsby’s tutorial guides folks to create their “first Gatsby site: a blog site…”
- Remix’s home page has a prominent Get Started call to action button that links to their Blog Tutorial.
Given that Markdown is a very popular document format for blogging, this gives us the opportunity to compare the performance of different site generators for this pervasive use case (highlighted on each generator’s documentation, as noted above).
Let’s see how each generator stacks up when consuming markdown files from the local project’s file system and building into a production-ready project.
(Order is alphabetical. Disclosure: I am the maintainer of Eleventy)
Benchmark Results
Times shown are in seconds. Lower is better.
| Markdown Files: | 250 | 500 | 1000 | 2000 | 4000 |
|---|---|---|---|---|---|
Astro 1.0.1 |
2.270 | 3.172 | 5.098 | 9.791 | 22.907 |
Eleventy 1.0.1 |
0.584 | 0.683 | 0.914 | 1.250 | 1.938 |
Gatsby 4.19.0-cli |
14.462 | 15.722 | 17.967 | 22.356 | 29.059 |
Hugo v0.101.0 |
0.071 | 0.110 | 0.171 | 0.352 | 0.684 |
Next.js 12.2.3 (JS routing) |
6.552 | 6.932 | 8.034 | 9.582 | 13.409 |
Next.js 12.2.3 (File routing) |
7.958 | 9.551 | 14.304 | 25.038 | 70.653 |
Remix 1.6.5 (File routing) |
2.876 | 8.258 | 46.918 | 349.125 | 1800 |
1.0.1.| Markdown Files: | 250 | 500 | 1000 | 2000 | 4000 |
|---|---|---|---|---|---|
Remix 1.6.5 (JS routing) |
0.868 | 0.834 | 0.891 | 0.887 | 0.901 |
Each run was repeated 3 times and the lowest/fastest time was selected. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5, 16 GB memory.
All of the code for this benchmark is fully open source and welcomes review.
Test Notes
For each generator sample I attempted to create a reduced project with the sole use case of processing markdown files. I opt-ed out of TypeScript when options were presented in various cli tools to do so. Output folders and framework specific cache folders were deleted before each run.
- For Astro I used the Blog example which uses a
pre-release of Astro 1.0. Updated August 10, 2022 Results were updated to Astro 1.0.1 after stable release! - For Eleventy I used
eleventy-base-blog, even though it has a few extra plugins and templates in play on top of the barebones core experience. - For Gatsby, I used
npm init gatsbywith Markdown support (not MDX). - For Hugo, I went through the Quickstart (skipping the theme)
- For Next.js I deleted a bunch of things out of the
blog-starterexample linked from the docs on Static Generation. - For Remix I went through the Blog tutorial but did not use a database. (Read the Remix addendum below).
I put out a Twitter poll to gauge how folks felt about project sizes. 1000 files was considered a Large project by 58.8% of voters, Medium for 36.8% of voters. Markdown samples borrowed from Sean C Davis’s SSG Build Performance Comparison repository. I have not yet added tests for Jekyll or Nuxt but I’m open to it!
Summary
Updated August 3, 2022 with notes about file-based routing of markdown in Remix and Next.js, as well as Astro MDX.
- Hugo remains the undisputed speed champ—no question about that.
- Eleventy was the fastest JavaScript-based generator.
- Generators that create per-page JavaScript bundles (for single page apps, primarily) are usually slower to build, unsurprisingly. Heavier pages aren’t exclusively slower for end users—they’re slower for developers too.
- File-based routing of markdown files in both Next.js and Remix is a bit slow! Take care to add the additional boilerplate routing code needed to load markdown files more efficiently.
- Read more on (nextjs.org) Advanced Features: Using MDX with Next.js and (remix.run) MDX and on the Remix addendum below.
- Astro was on-par with Next.js at mid-range (1k) and on-par with Gatsby at the upper-range (4k) of this benchmark.
- Jon Neal submitted a PR to do Astro markdown processing via the MDX plugin. I re-ran the tests but the data for Astro didn’t change in a statistically significant way (yet?). Likely more to come there, this feature is brand new!
- Updated August 10, 2022 Astro results were updated to use newly released stable version 1.0.1. This resulted in a ~25% improvement for Astro across the board 🏆
I would welcome a code review on the Remix site—it scaled so poorly that I suspect that I may have misconfigured something? I would be happy to update with corrections.- As noted in point 4 above, it is recommended for Remix projects by the Remix team to not use file-system based routing for
.mdxor.mdfiles. (Read the Remix addendum below).
- As noted in point 4 above, it is recommended for Remix projects by the Remix team to not use file-system based routing for
Bonus: npm install Benchmarks
Times shown are in seconds. Lower is better.
Show table of results
| Framework | npm install Time |
|---|---|
Astro 1.0.1 |
13.214 |
Eleventy 1.0.1 |
15.168 |
Eleventy 2.0.0-canary.14 |
7.195 |
Gatsby 4.19.0-cli |
68.516 |
Next.js 12.2.3 |
15.589 |
Remix 1.6.5 |
28.619 |
Each run was repeated 5 times and the lowest/fastest time was selected. npm cache clean --force was run before each to ensure a cold install. This result set was generated on a MacBook Air (M1, 2020), macOS Monterey 12.5, 16 GB memory.
Remix Addendum
Updated August 3, 2022 (this entire section is new)
I understand why Remix folks think this benchmark is unfair to Remix. I’ve heard lots of feedback and some of it has been decidedly unfriendly—but I do get it. I don’t really have any desire to wade into some of the larger debates about Remix or Jamstack on this post.
From my perspective, the benchmark encountered a build performance-related bug in how Remix—a request-time architected tool—performs a build-time precompilation of Markdown routes. It doesn’t have to be any more than a build-performance related bug. Once the bug is fixed (or if the approach is deprecated), I’ll make updates to this post.
For completeness and a bit more technical detail, the Remix team has asked me to highlight that it is not recommended to use MDX file-system based routing for .mdx or .md files. Specifically: do not put .md or .mdx files into your app/routes/ folder. Make sure you read/heed the warnings on the Remix MDX plugin documentation.
I maintain that markdown file-based routing is a useful feature and it seems likely that the Remix folks will be able to (at some point in the future) allow a peaceful coexistence of markdown routes without a heavy build-time compilation step. The demand clearly exists as many other frameworks do support it!
But until then the Remix team recommends to opt out of file-based routing for Markdown to bypass the costly build-time compilation step. You can use a scripted route file to load Markdown files from the file system in a different source directory (not app/routes/). You can see an implementation example from @ebey_jacob in the bench-framework-markdown repository.
One last note about Build Performance
“While you were waiting for your static site to build, distributed web infrastructure got really good. Break through the static.”—https://remix.run/
The only other thing I’d like to say here is that whether a tool spits out bundled JavaScript for runtime server deployment on the Edge or HTML in a static build for the CDN, whether it’s categorized as a Static Site Generator or a Site Generator or none of the above: all of the tools tested in this benchmark are performing a build step to generate code for deployment and can encounter build-related performance issues.
All that to say: I think we can agree that build performance is not a concern exclusive to static site generators! There will always be a healthy tension between request-time and build-time and the tools that make use of both to complement each other will likely end up being the most useful to developers.
Keep measuring and keep building—appreciate y’all!
58 Comments
@buhakmeh
The first time I used Hugo, I thought it was broken. I couldn't believe it was that fast.
@tvanantwerp
I think SSG goes against what Remix is trying to be. That said, still, wow...
@SCRWD
I would be interested to see where the ever-popular Jekyll sits in this test. I'm just itching for more reasons to convince management to let me switch everything to Eleventy haha. I still need some method for sharing and updating layouts and includes across multiple sites.
@SCRWD
Also, can you think of a way to compare the speed of templating between generators?
@zachleat
yeah, for sure—specifically this wasn’t attempting to use SSG, just sourcing markdown from the file system for a remix production build
@zachleat
so fast!
@zachleat
ah right on! maybe I can have a look at this on Monday 🙌🏻
@zachleat
The inputs vary a bit more across frameworks there—hmm
@zachleat
subscribe to this one github.com/zachleat/bench…
@BryceWrayTX
These numbers could be a big part of the reason why Astro essentially is dropping (or at least discouraging) pure Markdown in favor of MDX — which, at least in my own testing in dev mode, it handles much more quickly. 🤔
@zachleat
oh, maybe! @jon_neal was so quick to PR that and left some comments there: github.com/zachleat/bench…
@BryceWrayTX
Interesting! I should note that my tests involved only about 350 MDX files, albeit most of them very short (just quickies for the test), so his findings are probably much more meaningful than mine.
@nhoizey
I wonder if trends would be the same for 10k or 100k Markdown files… 😅
@zachleat
that might be my next test, then maybe on different hosting providers too 😅
@nhoizey
Hosting providers are interesting indeed! 👍
@m_yxnk
the other day i saw someone use hugo and i thought "why?". but now i'm ok with it after seeing the performance of remix (a "modern" metaframework) 😅
@zachleat
Hugo is good!
@m_yxnk
but 11ty is just better!
@zachleat
I mean, I agree but I am biased 😇
@tunetheweb
FYI the charts don’t work properly on mobile (iOS 15.5 using Safari in case it matters)
@bburgess_keys
2 things: 1. Woah, look at that Eleventy 2.0.0 install time!! 2. Lol Gatsby 😂
@j9t
And this one, because of the interesting generators comparison (even *if* @ZachLeat or I, as an @eleven_ty user, were biased—I think!): Which Generator Builds Markdown the Fastest?, by @zachleat, featured on @frontenddogma: zachleat.com/web/build-benc…
@jianjian086152
Remix: Thank you.
@AmeliasBrain
Enjoying how you handled the y-axis in the first figure, there. 😝
@zachleat
the noise I made when I learned that SVG could overflow: visible 😂
@AmeliasBrain
Next step in benchmarking would be the "add one new file & regenerate", I guess. Which of the generators have smarts to detect which content has changed, vs re-writing everything?
@papplegate
I don't know much, but I know I want the one that shoots up the whole page 😂
@zachleat
are you sure 😅
@papplegate
Nope. Two things I am sure of, death & paying my taxes .😅
@zachleat
yep it’s open source, linked up here I welcome the competition and will update it if a framework requests it!
@_furstenberg
And it’s pretty fast for Markdown as well. zachleat.com/web/build-benc… I currently use Hugo, but have created a version in Eleventy earlier.
@zachleat
Pushed a bunch of additions to this post after community feedback! 1️⃣ Added data for Next.js (file based routing) 2️⃣ A note about Astro and MDX (probably more to come there!) 3️⃣ A whole bunch of words on Remix! Deep link: zachleat.com/web/build-benc…
@brob
Who's pink here?
@zachleat
whew, I really don’t care to reopen this can-o-worms but you can look at the data table below the chart if it’s unclear (a bunch more conversation at )
@brob
Mostly just love that that line continues 😂
@PaoloRicciuti
Someone should definitely copy @remix_run code so that we can have more vertical lines.
@synecdokey
That probably came too late to change the initial perception. It's also arguably its Achille heel still zachleat.com/web/build-benc…
@nhoizey
Ça mérite un ajout dans le benchmark de @zachleat ! zachleat.com/web/build-benc…
@_magentix
J'ai répondu trop vite sans lire ^^
@ArnaudLigny
Excellente idée, ce sera plus qualifiant que faire des tests dans mon coin. Merci !
@nhoizey
🙏
@holger1411
Okay, seems you are right. the old 1.0 version is slower than Hugo: zachleat.com/web/build-benc… It is just the fastest JS based SSG But the latest canary 2.0 release is much faster than 1.0…so the race might be still interesting…
Frederic
@zachleat FYI, the chart in the article looks a bit odd on mobile. The line goes all the way to the top of the page. 😅
Charles Roper 🌻
@frederic @zachleat I quite like that effect.
Zach Leatherman :11ty:
@charlesroper @frederic I… well… it was intentional 😅
CJ Dunn
@zachleat Hex Franklin?
Zach Leatherman :11ty:
@cjtype tyght!! 🥰
Captain of the CSS Enterprise
@frederic @zachleat now that's what I call an edge case.
Šime Vidas
@zachleat Does font-weight: 900 serve as a fallback? Some browsers might attempt to synthesize that weight.
Zach Leatherman :11ty:
@simevidas do you see a fallback case in play? I have it listed in both my @font-face blocks and usage instances
Zach Leatherman :11ty:
@simevidas if you don’t set it, it’s implied to 400 which I believe would synthesize on an h1, via browser default bold. That said, I didn’t realize that font-synthesis browser support was so good now wow!
Roel Nieskens
@zachleat Subtle schmubtle! Looks good!
Zach Leatherman :11ty:
@pixelambacht thanks Roel!
@loige
and this one with an awesome graph zachleat.com/web/build-benc… :D
@joodaloop
Remix zachleat.com/web/build-benc…
@joodaloop
nvm zachleat.com/web/build-benc…
dan :html_energy:
@scrwd whoops, no Jekyll in here. Sorry!
Jon
@danleatherman no worries, I'm aware of this, it's fantastic and super grateful Zach took the time. Thanks for the link!