Category: Software

LSP, Clangd, NeoVim, Clangd: You Can Have It All. Even on Mac. Even when cross developing (ESP32 here).

Posted on by Robert Lipe

Edit: Aug 22. Necessary changes

Edit: Neovim/LSP keybindings added

I’ve come to accept that I’m an old-school vi user. With about forty years of muscle memory behind that sentence, I’ll say that I’m an effective vi user, but in today’s world of C++ overrides, sentence-long symbols, and huge lists of available methods, I _have_ felt the occasional pang of jealosy and tried to move beyond bare bone vi. I’ll then sit back down at the terminal two days later, completely forget about that snazzy new editor that I told myself I’d learn, and immediately be back to a couple of iTerms with ack-grep just from habit. Making matters worse for myself, most of the time these days I work on ESP32 projects, so I’m dealing with cross compilers. Well, I’m not unhappy about that – having basically the same compiler on my embedded device as I do on my desktop is actually pretty sweet, but they’re always just different enough to be annoying. One difference that surfaces is that clangd-like services for Language Server Protocol (LSP) are usually just out of reach. One recent day when alternating between fighting CLion and vi (OK, it was probably actually vim; though I have flirtations with nvim.) I set out to win this battle.

It turns out there’s a slew of annoying behaviour you must overcome to get niceties to work. As is my style, I don’t declare this is the One True Way or even the Only Way. It’s just the journey that I took and I’m sharing it in the hopes that it saves you a few hours in your own journey. I’ll focus first on CLion and ESP32 because from there, “winning” with nvim is just an exercise in persistence. If it’s your goal to ignore CLion and focus on nvim or other clangd consumers, the steps to skip should be self-evident.

CLion claims to be able to use a platformio.ini. This is at best partially true. It can read the list of [env:] projects and give you a list of -e options that it will pass when shelling out to the (already slow and often broken) Platformio ‘pio’ command. By default, when given my (admittedly non-trivial, but hardly back-breaking) platformio.ini of 53 environments, we’re met with a red “Index 0 out of bounds for length 0”. No hint is given on what array was being indexed, what source file might be responsible, or even what language might have issued this error. All #included files are red squigglies because it can’t find them. No symbols have hover text unless they’re declared in the file you’re looking at, and syntax highlighting is about as effective as regex highlighting ever is on C++. Good job, Jetbrains.

Populate CLion’s shadow build system:

$ pio project init --ide clion

Resolving demo dependencies... (demo is the name of one environment in my platformio.ini that I'll use throughout)
Already up-to-date.
Updating metadata for the clion IDE...
Project has been successfully updated!

Restart CLion to see the changes.

$ open -a clion . (or whatever pointy clicky equivalent gives you joy)

It’s my experience, after probably a hundred restarts while developing this recipe, that this will fail on odd numbered invocations and pass on even numbered invocations… (Did I yet bitterly say “Good job, Jetbrains?”)

Now the bottom will show a blue progress bar during an ‘importing’ step that for me, takes about four minutes (!!) to complete. I have no idea why it takes long to analyze than it takes to completely build one or two of our targets, but I’m just a lowly bitslinger.

Restart CLion (twice) to see the changes.

Now it’ll still give the error and it still won’t convince you that it’s read platformio.ini, but it’s apparently necessary ritual sacrifice.

Build the Compilation “database”.

$ pio run -t compiledb -e demo

[ ... ]

Building in release mode
Building compilation database compile_commands.json

========================= [SUCCESS] Took 34.44 seconds =========================
Environment    Status    Duration
-------------  --------  ------------
demo           SUCCESS   00:00:34.439

========================= 1 succeeded in 00:00:34.439 =========================

Note that it will download and install all the packages required for all the builds (which is why I’m restricting it to ‘-e demo’ here or I’d be typing this until the cows came home. (Where do the cows GO anyway?) On my project, that’s about 20 libraries times 53 -e environments, so even though it’s not actually BUILDING much of anything, it’s still an exercise in persistence.

You now have a shiny new `compile_commands.json` file that describes what it would have invoked to build the software. Well, it doesn’t include the critical linking step, so it’s still not REALLY helpful if you’re trying to do what a ‘pio run’ would do, but it would tell you what the first 99% of a build will do. In my case, it’s 830 lines of blindingly dumb JSON that makes no attempt to factor out redundancy. These files ARE handy if you want to see what flags really really get applied to your build without looking at `-v`.

Purge the old CLion config

rm -fr .idea 

Now tell CLion to open that directory. It will recognize `compile_commands.json` and open it. The bottom of your tools menu will now have an option for Compilation Database. Your source tree will now show your source AND the zillion files in `~/.platformio/packages/framework-arduinoespressif32/libraries/*/src` that are ACTUALLY built. It also now knows where all your includes are as well as the system includes.

Notice that you can do OK living a bit of a lie here. The include paths for a lot of the system stuff actually change slightly depending if you’re targeting LX6 (ESP32-Classic), LX7 (ESP32-S2 or ESP32-S3) or RISC-V (everything else), but unless you’re working at the very bottom of the stack and NEED to know which source is used for the system stuff, it’s probably OK.  If it matters, you can repeat the pio run -t compiledb with a different -e and then reload the compilation database.

So this is great, right?

No. This is OK for editing in CLion, but you can’t actually BUILD in CLion this way. Despite giving it agonizing details of every invocation of g++, every one of the hundreds of `-I` flags and all that other stuff (look in `compile_commands.json`, seriously!) it won’t actually build code. You still have to open a window and `pio run -t upload -e foo` for that. There may be workarounds for that, but we’ll come back because vim is where work gets done while CLion is nice for fix-ups and clang warnings. Let’s forge on.

We have a great compile_commands.json, we can surely now rely on LSP support in nvim, right?

No.

We’re thwarted by many things. It seems that, despite having these agonizingly detailed roadmaps describing how to compile (almost) everything, both `clangd` and CLion have their own internal whitelists of toolchains. If your g++ binary isn’t whitelisted, it will silently fail and then try to invoke the native compiler. So the first time you hit, say `<vector>` or `<cstdio>` everything hits the fan because your MacOS or native Linux `/bin/g++`’s headers are in the “wrong” place and they call things that aren’t in the `-I` paths that are given. For MacOS, `_ansi.h` seems to be the poison pill, but if it weren’t that file, it would be something else.

I’ll skip ahead about two hours of debugging, but the solution lies in creating a .clangd in the root of your project’s directory to add the flags that you need and to strip the flags that nvim tries to add to the build that would choke your cross compiler, like anything starting with -m because you’re presumably compiling for a different machine architecture.

CompileFlags:
  Remove:
    - -m*
    - -fno-tree-switch-conversion
    - -fstrict-volatile-bitfields
    - -Wno-frame-address

  Add:
    - -nostdlibinc
    - -nostdinc++

I’ll fast-forward you through another two hours, though, and tell you that the `clangd` on Mac is so hell-bent on developing for Mac that it will ignore the above and STILL add things to its own include path that are toxic to the cross compiler. This is because Apple’s `/usr/bin/clangd` is a stripped-down, lobotomized version tied to Xcode that lacks standard features and stubbornly injects Apple-specific flags. The solution to this is to install `clangd` from Homebrew (`brew install llvm`) and then add a wrapper file (remember to mark it executable) in a directory in your `$PATH` that’s before the system binary. I already had `~/.local/bin/` in my PATH, so I’ll just add that.

My new ~/.local/bin/clangd:

#!/bin/sh

# Find the real clangd by listing ALL instances and skipping this wrapper
#REAL_CLANGD=$(which -a clangd | grep -v "\.local/bin/clangd" | head -n 1)

REAL_CLANGD=/opt/homebrew/opt/llvm/bin/clangd

# Execute it with the query-driver argument pointing to the global PlatformIO directory
exec "$REAL_CLANGD" --query-driver="$HOME/.platformio/packages/**/bin/*g++*,$HOME/.platformio/packages/**/bin/*gcc*" "$@"

Testing this mess without the editor

You’ll know this is in the game when you can do something like:

$ clangd --background-index --check=src/main.cpp 2>&1 | less

(Lipe-pro-tip: `–background-index` is generally a good flag to add in your editor setups so `clangd` can eagerly cache your project and make cross-file navigation instant.)

and see the  presence of:

I[01:22:30.648] Loading compilation database...
I[01:22:30.667] Loaded compilation database from ...
I[01:22:30.668] Compile command from CDB is:...

…and output that ends like:

I[01:23:24.137] All checks completed, 6 errors

It’s not important that it be zero. It’s important that it not be so many that clangd aborts. In fact, it’s even normal to have hundreds of lines like

E[01:23:24.130] IncludeCleaner: Failed to get an entry for resolved path : No such file or directory

or

E[01:25:02.807]     tweak: ExpandDeducedType ==> FAIL: Could not deduce type for 'auto' type

Despite these starting with “E:” and LOOKING like errors, this is expected.

** Note: if you tinker much with .clangd, it seems that a rm -fr .cache/clangd is sometimes needed. Since it’s not clear WHEN it’s needed and it tends to be pretty fast to just rebuild, if you’re working on this, just keep a rm -fr .clangd/cache ; nvim .clangd in your recall buffer.

NOW, you’ve slain the final boss between you and glorious nvim assisted by clangd via LSP.

<screenshot goes here>

Now, tab completion, hover, and auto clang-tidy all work sensibly. Victory!

But nvim screws up copy-paste?!?! Yeah, well…future battles to fight. *(Though if you want to skip the wrapper script entirely in Neovim, you can actually pass `–query-driver` directly in your `nvim-lspconfig` setup for `clangd` using the `cmd` table!)*

But I want to BUILD in CLion!

Since we bypassed CLion’s native PlatformIO integration (because it was choking on our 53 environments) and loaded the project as a bare Compilation Database, CLion doesn’t actually know *how* to compile your code. It just knows how to parse it. If you hit the hammer icon, nothing good happens.

To fix this, we need to teach CLion how to shell out to `pio`. We do this using Custom Build Targets.

1. Go to **Preferences / Settings -> Build, Execution, Deployment -> Custom Build Targets**.
2. Click the `+` to add a new target and call it something creative like “PIO Build”.
3. Next to the “Build” field, click the `…` to add an External Tool. 
4. Click `+` to add a new External Tool.

   – **Name**: `pio run demo`
   – **Program**: `pio` (or the full path like `~/.platformio/penv/bin/pio` if CLion’s `$PATH` is lacking)
   – **Arguments**: `run -e demo` (or whatever environment you want to actually build)
   – **Working directory**: `$ProjectFileDir$`

5. (Optional but helpful) Do the exact same thing for the “Clean” tool, passing `run -t clean -e demo` as the arguments
6. Save your new target.

Now, go to the Run/Debug Configurations dropdown in the toolbar (or **Run -> Edit Configurations**).

1. Click `+` and select **Custom Build Application**.
2. Set the **Target** to the “PIO Build” target you just created.
3. You can leave the Executable blank if you just want to build, or point it to your compiled `.elf` file if you have ambitions of setting up an Embedded GDB Server later.

Now you can smash that hammer icon or hit `Cmd+F9` (or your OS equivalent), and CLion will obediently shell out to PlatformIO, build your project, and dump the output in the console window.  If the native Platformio support worked, that’s all it would do for you anyway.

You’ve finally got the best of both worlds: CLion’s shiny graphical UI, proper code insight and navigation powered by your compilation database, and the actual build process correctly handled by PlatformIO. And when you’re ready to get real work done, your `.clangd` and wrapper script mean Neovim is sitting right there in your terminal, ready to go with full LSP support.

Take a victory lap. (And/or a beverage of your choosing.) You’ve earned it.

 

Shifting sands. Article changes.

I admittedly don’t use this configuration every day. I’d noticed that it had quit working, but I also wasn’t certain that I’d tested it on this exact configuration so I let it coast until that little rock in my shoe annoyed me enough to tackle it. Still, I knew my approach was sound but I couldn’t suss it out. Eventually, I asked my friend Gemini (Antigravity) to help. I’m leaving the above unedited because that probably works for some configurations. Here’s what we discovered.

  • On this system, my $HOME is a symlink to an external SSD. (Thank you, Apple for your extortionistic storage prices.) My ~/.local/bin/clangd suffered from a one-two punch of overly strict matching and an update to PlatformIO moving the internal platform file.• What changed: We pointed REAL_CLANGD away from Homebrew’s LLVM and directly to the esp-clangd binary installed by EIM (~/.espressif/tools/esp-clangd/…).
    • The sneaky fix: EIM actually splits esp-clangd and esp-clang into two completely separate directories. This means the clangd binary doesn’t know where its own built-in headers are! We had to append –resource-dir=”~/.espressif/tools/esp-clang/…/lib/clang/21″ so it
    could find basic things like stddef.h. While we are here, we tweaked things so we only apply all this cleverness if we have a .platformio.ini containing the word “espressif” (so nvim on native projects still work) AND we dig around in the espressif tree to find the most recent esp-clangd.

    Updated file:

    $ cat ~/.local/bin/clangd


#!/bin/sh

# 1. Detect if we are inside an ESP32 PlatformIO project

if [ -f "platformio.ini" ] && grep -qi "espressif" "platformio.ini"; then

    # 2. Dynamically find the newest esp-clangd and resource dirs
    ESP_CLANGD_DIR=$(ls -rd "$HOME"/.espressif/tools/esp-clangd/* 2>/dev/null | head -n 1)

    REAL_CLANGD="$ESP_CLANGD_DIR/esp-clangd/bin/clangd"
    ESP_CLANG_DIR=$(ls -rd "$HOME"/.espressif/tools/esp-clang/* 2>/dev/null | head -n 1)
    RESOURCE_DIR=$(ls -rd "$ESP_CLANG_DIR"/esp-clang/lib/clang/* 2>/dev/null | head -n 1)

    # 3. If the ESP tools exist, use them
    if [ -x "$REAL_CLANGD" ] && [ -d "$RESOURCE_DIR" ]; then

exec "$REAL_CLANGD" \

     --query-driver="$HOME/.platformio/packages/**/bin/*g++*" \
     --resource-dir="$RESOURCE_DIR" \
     "$@"
    fi
    # If the ESP tools were missing, it safely falls through to the system default below
fi

# 4. We are NOT in an ESP project (or ESP tools are missing).
# Fall back to standard desktop C++ clangd.
# Find the first clangd that IS NOT this wrapper script itself.
SYSTEM_CLANGD=$(which -a clangd | grep -v "\.local/bin/clangd" | head -n 1)

if [ -n "$SYSTEM_CLANGD" ]; then
    exec "$SYSTEM_CLANGD" "$@"
else
    echo "No system clangd found." >&2
    exit 1
fi

if [ -n "$SYSTEM_CLANGD" ]; then
    exec "$SYSTEM_CLANGD" "$@"
else
    echo "No system clangd found." >&2
    exit 1
fi
  • NeoVIM (AstroNVIM?) changed direction, requiring tweaks to the Neovim Config (~/.config/nvim/lua/plugins/astrolsp.lua)
    • What changed: We explicitly set cmd = { “/Users/robertlipe/.local/bin/clangd”, … } instead of just “clangd”.
    • Why it broke your previous setup: AstroNvim recently integrated Mason very tightly. If you just ask for “clangd”, Mason intercepts your $PATH and silently launches its own vanilla LLVM clangd. This completely bypassed your wrapper script, dropping your –query-driver argument, and putting you back to square one. Hardcoding the absolute path forces Neovim to respect your wrapper. Solution:Add a section under “servers…pyright” stanza

        servers = {
      -- "pyright"
    },

        -- customize language server configuration options passed to `lspconfig`
    ---@diagnostic disable: missing-fields
    config = {
      clangd = {
        cmd = {
          "/Users/robertlipe/.local/bin/clangd",
          "--background-index",
        },
        capabilities = { offsetEncoding = "utf-8" },
      },
    },

        -- customize how language servers are attached
    handlers = {
      -- a function without a key is simply the default handler, functions take two parameters, the server name and the configured options table for that server
  • I moved to C++26 and this made PlatformIO grumpy – Changes to .clangd required
    A few months ago, I bumped my minimum platform requirements to C++26 (because I can). PlatformIO adds its own –std= lines into the build and depending on the time of day, that can come before or after my own entry. GCC parses arguments left to right, so the last one standing ‘wins’. When calling clang-format, it was getting reset, so my
    static_assert(__cplusplus > 202002L, "This project requires C++26");
    was sending the parse down in flames.
    • What changed: We keep the Remove: block to strip out the GCC-specific optimization flags that cause warnings. However, we were able to delete -nostdlibinc and -nostdinc++. We have to add back the -std=gnu++2b flag.
    • Why it’s better: Because you’re now using Espressif’s custom clangd, it natively understands the Xtensa target. You no longer have to manually nuke the Apple standard headers because Espressif’s parser doesn’t confusingly fall back to macOS Mach-O headers anymore!You officially have the ultimate setup. No cosmetic #error in limits.h, native understanding of .iram1 ELF sections, and perfect C++ standard library resolution!

 

Appendix: Neovim/LSP Keybindings (the reason we do this…) 

Since this is largely documentation for myself, here, at no extra cost to you, is a subset of the keybindings that are most useful.

Feature Keybinding Lua Function/Command What it does
Hover Documentation K vim.lsp.buf.hover() Pops up a floating window showing the comment block, type info, and documentation for the symbol under your cursor.
Go to Definition gd vim.lsp.buf.definition() Jumps straight to where the variable, function, or class is defined.
Go to Declaration gD vim.lsp.buf.declaration() Jumps to the header declaration (very handy for C++ header/source splits).
Document Symbols gO vim.lsp.buf.document_symbol() Lists all methods, classes, and variables in the current file so you can fuzzy-search through them.Code Actions
Code Actions gra vim.lsp.buf.code_action() Fixes missing includes, implements virtual methods, or applies automated refactors.
Smart Rename grn vim.lsp.buf.rename() Renames a symbol safely across your entire project workspace.
Type Definition grt vim.lsp.buf.type_definition() Jumps to the underlying type definition of a variable.
Go to Implementation gri vim.lsp.buf.implementation() Jumps to subclasses or concrete implementations of a virtual interface.
Find References grr vim.lsp.buf.references() Scans your entire project and lists every place that symbol is used.
       
       

 

There is a more comprehensive version of Neovim LSP Keybindings

When portability decisions can be “fuzzy”. Defend your design decisions.

Posted on by Robert Lipe

I just finished watching Fast and Small C++ on YouTube. The title was a tad misleading for my taste, as it was a case study on a few cases instead of a guide on creating the same, but here we are.

The speaker delivers a (very) few examples but does a case study on three implementations of the Small String Optimization in MSVC, GNU Libg++, and Clang’s libc++. I started writing this as a comment on YouTube but figured it was better to write it here (where nobody will read it) instead of a long post on YouTube (where nobody will read it).

The questioners touched on an interesting topic that probably doesn’t get enough attention. Implementations of systems software, probably more than application software, almost always have the ability to make tradeoffs where you can negotiate.

If you KNOW what generations of hardware you run on (perhaps you don’t care about 36-bit, middle-endian, context switching on an RCA 1802, output to clay marble tablets, or whatever), **and** you’re willing to throw in some static_asserts and a run-time test suite, it’s often OK to just plain not care if your pointer address == your SSO string length (or whatever example the questioner called out) because you’re literally writing the rule book, and you can just declare that to not be a case you care about.

* Unaligned accesses are slow? Don’t care.
* Unaligned accesses crashing? Don’t care.
* Your memory bits count from zero, up, and your data bits count from 64, down? Don’t care.

Repeat this pattern. It might be long. You might get a bonus for documenting them, asserting them, etc., so that the next person that is trying to move this from a data center to a $0.09 CH32V003 can at least know there are dragons ahead. That team, operating under different constraints, is then free/encouraged/required to revisit those decisions. It’s centralizing those decisions that’s the key.

If you’re writing code that runs inside a data center on hardware you control, you have incredible control over where that code runs and what compromises you have to make. Yes, it’d be polite if you could give an unfortunate time-traveler the courtesy of a failed compile-time assertion for such failures, but if you can save Google, Facebook, OpenAI, or whomever 5% of a runtime (which may correspond to the electricity bill of a small city, and that’s not even much of an exaggeration), architectural purity for compatibility with mercury tube memory RAM cells may not matter when you can change one line in a config file and say, “Software built with flags X doesn’t run in data center nodes Y and Z. Don’t bother me if you try,” because you walked down the (virtual) halls and blessed it with the compiler team, the hardware team, the OS team, the CPU team, etc. That’s actually often OK. I’m guessing that if you’re the Microsoft STL team, it’s within your right to not care much if your floating point library screams on an Arduino that’s emulating IEEE754 in software anyway, for example. The team implementing STL for a Google data center may not care about SSO in the way that a similar-sounding team implementing STL for ChromeOS may have to.

Some of these bit-tweaking decisions can have HUGE ripple impacts through modern large systems. If your SSO can hold 23 bytes instead of 7 but costs you 8% compute overhead but saves you $$$$ in VM/paging overhead because you’ve profiled your last zillion transactions and you KNOW that 89% of your strings are 19 bytes long or less, you can look like a hero. Maybe your cores penalize you heavily for branches but make speculation free. Maybe that all falls apart when your loops get automatically vectorized to completely different packages and not just different execution units within the same core. These things can all lead to wildly different core data structures and implementations.

If you’re implementing the std::string that goes with a specific compiler for a specific class of computing environments (STM32 just has different rules than z/OS). This is OK.) It’s fine to declare that the rules you need to play by are different. As a courtesy, document or at least announce those compromises in a test suite or at compile time.

Don’t go nuts with this liberty. If you make a web server on an ESP32 for an air quality device running in a single home and can save a clock cycle and make it fail silently, working on an ESP32-H2 when it works on an ESP32-C3 [1] (in a way that nobody actually cares), you’re just a psychopath and deserve to have no friends. Don’t do that. You need to be able to justify such decisions on a really large scale, IMO. I worked on the URL parser in GPSBabel that probably parses many hundreds of URLs a year, scattered across many different computers. I also made some changes to the URL parser at Google.com. They have very, very different constraints.

I was in systems software for a long time, and I’d like one of the key takeaways from this talk to be that even if your goal is to make something as “simple” as a std::string be 24 bytes long [2], there are different approaches with different tradeoffs and different cliffs when the constraints are violated. (For example, there’s a LOT of value in having the SSO buffer start at offset zero of the structure. The equivalent of an unprototyped call to puts() on a std::string (eek!) has some chance of having at least some of the string be recognizable.

Large companies these days have access to total systems measurements we dreamed of in the past. “Let’s build the top 1000 GitHub projects on a toolchain implemented to measure X and see what the average string size is” is incredibly powerful for helping decide if the SSO length should be 7, 8, 15, 23, or (gasp) more. We have the tools these days to measure this stuff (before AND after) and to constrain the worlds where the code runs to fit those resulting decisions. Maybe the SSO values can be different for a Z80 and a Google Datacenter, where every opcode from boot to recycling is controlled by one software stack.

Signed,
Worked in really large companies.

[1] I’m not even totally sure that such a thing is possible. My point is that it’s probably not wise. There should be mountains of hair saved when shaving such yaks, not individual hairs. You should be able to point to receipts of shipping those bales of hair and include projections of shipping costs for future generations of yak hair.

[2] It’s not defended that “24” was even the best choice, though it was the choice of all three implementations. “As small as possible” has strong merit, but there are a lot of apps that deal in strings that don’t quite fit into SSO buffers. There’s also a large number of programs that won’t possibly care about the difference between SSO breaking points being 7 or 28. This is probably not why your CRUD app is slow. If you’re going to care about this, be sure it’s defensible.