05/06/23 – this is written in anger at 02:20a. I plan to come back and check/format this .
I’ve come to accept that I’m an old-school vi user. With about forty years of muscle memory behind that sentence, I’ll say that I’m an effective vi user, but in today’s world of C++ overrides, sentence-long symbols, and huge lists of available methods, I _have_ felt the occaisional pang of jealosy and tried to move beyond bare bone vi. Then I’ll sit back down at the terminal two days later, completely forget about that snazzy new editor that I told myself I’d learn, and immediately be back to a couple of iTerms with ack-grep just from habit.To make matters worse for myself, most of the time these days I work on ESP32 projects, so I’m dealing with cross compilers. Well, I’m not unhappy about that – having _basically_ the same compiler on my embedded device as I do on my desktop is actually pretty sweet, but they’re always just different enough to annoying. One difference that surfaces is that clangd-like services for Language Server Protocol (LSP) are usually just out of reach. One recent day when alternating between fighting CLion and vi (OK, itt was probably actually vim; though with flirtations with nvim.) I set out to win this battle.
It turns out there’s a slew of annoying behaviour you must overcome to get niceties to work. As is my style, I don’t declare this is the One True Way or even the Only Way. It’s just the journey that I took and I’m sharing it in the hopes that it saves you a few hours in your own journey. I’ll focus first on CLion and ESP32 because from there, “winning” with nvim is just an exercise in persistence. If it’s your goal to ignore CLion and focus on nvim or other clangd consumers, the steps to skip should be self-evident.
CLion claims to be able to use a platformio.ini. This is at best partially true. It can read the list of [env:] projects and give you a list of -e optinos that it will pass when shelling out to the (already slow and often broken) Platformio ‘pio’ command. By default, when given my (admittedly non-trivial, but hardly back-breaking) platformio.ini of 53 environments, we’re met with a red “Index 0 out of bounds for length 0”. No hint is given on what array was being indexed, what source file might be responsible, or even what language might have issued this error. All #includes are red squigglies because it can’t find them. No symbols have hover text unless they’re declared in the file you’re looking at, and syntax highlighting is about as effective as regex highlighting ever is on C++. Good job, Jet Brains.
Populate CLion’s shadow build system:
$ pio project init --ide clion Resolving demo dependencies... (demo is the name of one environment in my platformio.ini that I'll use throughout) Already up-to-date. Updating metadata for the clion IDE... Project has been successfully updated!
Restart CLion to see the changes.
$ open -a clion . (or whatever pointy clicky equivalent gives you joy)
It’s my experience, after probably a hundred restarts while developing this recipe, that this will fail on odd numbered invocations and pass on even numbered invocations… (Did I yet say “Good job, Jet Brains?”)
Now the bottom will have will show a blue progress bar during an ‘importing’ step that for me, takes about four minutes (!!) to complete. I have no idea why it takes long to analyze than it takes to completely build one or two of our targets, but I’m just a lowly bitslinger.
Restart CLion (twice) to see the changes.
Now it’ll still give the error and it still won’t convince you that it’s read platformio.ini, but it’s apparently ncessary ritual sacrifice.
Build the Compliation “database”.
$ pio run -t compiledb -e demo [ ... ] Building in release mode Building compilation database compile_commands.json ========================= [SUCCESS] Took 34.44 seconds ========================= Environment Status Duration ------------- -------- ------------ demo SUCCESS 00:00:34.439 ========================= 1 succeeded in 00:00:34.439 =========================
Note that it will download and install all the packages required for all the builds (which is why I’m restricting it to ‘-e demo’ here or I’d be typing this until the cows came home. (Where do the cows GO anyway?) On my project, that’s about 20 libraries times 53 -e environments, so even though it’s not actually BUILDING much of anything, it’s still an exercise in persistence.
You now have a shiny new `compile_commands.json` file that describes what it would have invoked to build the software. Well, it doesn’t include the critical linking step, so it’s still not REALLY helpful if you’re trying to do what a ‘pio run’ would do, but it would tell you what the first 99% of a build will do. In my case, it’s 830 lines of blindingly dumb JSON that makes no attempt to factor out redundancy. These files ARE handy if you want to see what flags really really get applied to your build without looking at `-v`.
Purge the old CLion config
rm -fr .idea
Now tell CLion to open that directory. It will recognize `compile_commands.json` and open it. The bottom of your tools menu will now have an option for Compilation Database. Your source tree will now show your source AND the zillion files in `~/.platformio/packages/framework-arduinoespressif32/libraries/*/src` that are ACTUALLY built. It also now knows where all your includes are as well as the system includes.
Notice that you can do OK living a bit of a lie here. The include paths for a lot of the system stuff actually change slightly depending if you’re targeting LX6 (ESP32-Classic), LX7 (ESP32-S2 or ESP32-S3) or RISC-V (everything else), but unless you’re working at the very bottom of the stack and NEED to know which source is used for the system stuff, it’s probably OK. If it matters, you can repeat the pio run -t compiledb with a different -e and then reload the compilation database.
So this is great, right?
No. This is OK for editing in CLion, but you can’t actually BUILD in CLion this way. Despite giving it agonizing details of every invocation of g++, every one of the hundreds of `-I` flags and all that other stuff (look in `compile_commands.json`, seriously!) it won’t actually build code. You still have to open a window and `pio run -t upload -e foo` for that. There may be workarounds for that, but we’ll come back because vim is where work gets done while CLion is nice for fixups and clang warnings. Let’s forge on.
We have a great compile_commands.json, we can surely now rely on LSP support in nvim, right?
No.
We’re thwarted by many things. It seems that, despite having these agonizingly detailed roadmaps describing how to compile (almost) everything, both `clangd` and CLion have their own internal whitelists of toolchains. If your g++ binary isn’t whitelisted, it will silently fail and then try to invoke the native compiler. So the first time you hit, say `<vector>` or `<cstdio>` everything hits the fan because your MacOS or native Linux `/bin/g++`’s headers are in the “wrong” place and they call things that aren’t in the `-I` paths that are given. For MacOS, `_ansi.h` seems to be the poison pill, but if it weren’t that file, it would be something else.
I’ll skip ahead about two hours of debugging, but the solution lies in creating a .clangd in the root of your project’s directory to add the flags that you need and to strip the flags that nvim tries to add to the build that would choke your cross compiler, like anything starting with -m because you’re presumably compiling for a different machine architecture.
CompileFlags: Remove: - -m* - -fno-tree-switch-conversion - -fstrict-volatile-bitfields - -Wno-frame-address Add: - -nostdlibinc - -nostdinc++
I’ll fast-forward you through another two hours, though, and tell you that the `clangd` on Mac is so hell-bent on developing for Mac that it will ignore the above and STILL add things to its own include path that are toxic to the cross compiler. This is because Apple’s `/usr/bin/clangd` is a stripped-down, lobotomized version tied to Xcode that lacks standard features and stubbornly injects Apple-specific flags. The solution to this is to install `clangd` from Homebrew (`brew install llvm`) and then add a wrapper file (remember to mark it executable) in a directory in your `$PATH` that’s before the system binary. I already had `~/.local/bin/` in my PATH, so I’ll just add that.
My new ~/.local/bin/clangd:
#!/bin/sh # Find the real clangd by listing ALL instances and skipping this wrapper #REAL_CLANGD=$(which -a clangd | grep -v "\.local/bin/clangd" | head -n 1) REAL_CLANGD=/opt/homebrew/opt/llvm/bin/clangd # Execute it with the query-driver argument pointing to the global PlatformIO directory exec "$REAL_CLANGD" --query-driver="$HOME/.platformio/packages/**/bin/*g++*,$HOME/.platformio/packages/**/bin/*gcc*" "$@"
Testing this mess without the editor
You’ll know this is in the game when you can do something like:
$ clangd --background-index --check=src/main.cpp 2>&1 | less
(Lipe-pro-tip: `–background-index` is generally a good flag to add in your editor setups so `clangd` can eagerly cache your project and make cross-file navigation instant.)
and see the presence of:
I[01:22:30.648] Loading compilation database... I[01:22:30.667] Loaded compilation database from ... I[01:22:30.668] Compile command from CDB is:...
…and output that ends like:
I[01:23:24.137] All checks completed, 6 errors
It’s not important that it be zero. It’s important that it not be so many that clangd aborts. In fact, it’s even normal to have hundreds of lines like
E[01:23:24.130] IncludeCleaner: Failed to get an entry for resolved path : No such file or directory
or
E[01:25:02.807] tweak: ExpandDeducedType ==> FAIL: Could not deduce type for 'auto' type
Despite these starting with “E:” and LOOKING like errors, this is expected.
** Note: if you tinker much with .clangd, it seems that a rm -fr .cache/clangd is sometimes needed. Since it’s not clear WHEN it’s needed and it tends to be pretty fast to just rebuild, if you’re working on this, just keep a rm -fr .clangd/cache ; nvim .clangd in your recall buffer.
NOW, you’ve slain the final boss between you and glorious nvim assisted by clangd via LSP.
<screenshot goes here>
Now, tab completion, hover, and auto clang-tidy all work sensibly. Victory!
But nvim screws up copy-paste?!?! Yeah, well…future battles to fight. *(Though if you want to skip the wrapper script entirely in Neovim, you can actually pass `–query-driver` directly in your `nvim-lspconfig` setup for `clangd` using the `cmd` table!)*
But I want to BUILD in CLion!
Since we bypassed CLion’s native PlatformIO integration (because it was choking on our 53 environments) and loaded the project as a bare Compilation Database, CLion doesn’t actually know *how* to compile your code. It just knows how to parse it. If you hit the hammer icon, nothing good happens.
To fix this, we need to teach CLion how to shell out to `pio`. We do this using Custom Build Targets.
1. Go to **Preferences / Settings -> Build, Execution, Deployment -> Custom Build Targets**.
2. Click the `+` to add a new target and call it something creative like “PIO Build”.
3. Next to the “Build” field, click the `…` to add an External Tool.
4. Click `+` to add a new External Tool.
– **Name**: `pio run demo`
– **Program**: `pio` (or the full path like `~/.platformio/penv/bin/pio` if CLion’s `$PATH` is lacking)
– **Arguments**: `run -e demo` (or whatever environment you want to actually build)
– **Working directory**: `$ProjectFileDir$`
5. (Optional but helpful) Do the exact same thing for the “Clean” tool, passing `run -t clean -e demo` as the arguments
6. Save your new target.
Now, go to the Run/Debug Configurations dropdown in the toolbar (or **Run -> Edit Configurations**).
1. Click `+` and select **Custom Build Application**.
2. Set the **Target** to the “PIO Build” target you just created.
3. You can leave the Executable blank if you just want to build, or point it to your compiled `.elf` file if you have ambitions of setting up an Embedded GDB Server later.
Now you can smash that hammer icon or hit `Cmd+F9` (or your OS equivalent), and CLion will obediently shell out to PlatformIO, build your project, and dump the output in the console window. If the native Platformio support worked, that’s all it would do for you anyway.
You’ve finally got the best of both worlds: CLion’s shiny graphical UI, proper code insight and navigation powered by your compilation database, and the actual build process correctly handled by PlatformIO. And when you’re ready to get real work done, your `.clangd` and wrapper script mean Neovim is sitting right there in your terminal, ready to go with full LSP support.
Take a victory lap. (And/or a beverage of your choosing.) You’ve earned it.