Setting Up a Local Ollama Copilot via LSP

I am quite interested in running AI offline. Thus I really like Ollama, and have added automatic failover from ChatGPT to a local AI to my little terminal llm tool cll (get it on Github at akirk/cll).

As a developer, an important local gap for me was Github Copilot. Its function of autocomplete on steroids is really powerful in my day to day work and speeds up my development a lot.

Now, how can you get this offline? Mostly, search engines point to solutions that involve Visual Studio Code extensions, for example Continue and lots of other dependencies.

LSPs are independent of IDEs

But why should this involve IDE extensions? With the concept of LSPs (read LSP: the good, the bad, and the ugly to learn how LSPs work), and the existence of LSP-Copilot, this should be independent of IDEs. And I personally use Sublime Text.

And indeed, it does work just on that basis: using the go proxy ollama-copilot by Bernardo de Oliveira Bruning.

But for me it didn’t work out of the box. Thus, I’d like to share the steps that got this working for me. I use macOS.

Steps to get it running

First, follow the install instructions for Ollama and ollama-copilot. This puts the go binary in ~/go/bin/ollama-copilot

Then, change the settings for lsp-copilot and add "proxy": "127.0.0.1:11435" (this is the default local port).

Now, you also need to address the certificate situation. I use mkcert which you can install with homebrew using

brew install mkcert

Follow the instructions to install its root cert. We need a certificate that covers two Edit: three hosts, so run

cd ~/go/bin/; mkcert api.github.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com

which gives you two files with which you can now now start the proxy:

~/go/bin/ollama-copilot -cert ~/go/bin/api.github.com+2.pem -key ~/go/bin/api.github.com+2-key.pem

Finally, you need to add one more thing to the lsp-copilot config JSON. First find out the location of the root cert: echo $(mkcert -CAROOT)/rootCA.pem and add an env section there (see this FAQ), for me it’s:

"env": {
	"NODE_EXTRA_CA_CERTS": "~/Library/Application Support/mkcert/rootCA.pem"
},

This made it work for me. Edit: It seems a bit erratic. For me it works most reliably if you start ollama-copilot first, and only then Sublime Text. You can see the proxy at work through its output in the terminal.

2024/11/15 16:04:08 request: POST /v1/engines/copilot-codex/completions
2024/11/15 16:04:12 response: POST /v1/engines/copilot-codex/completions 200 4.744932083s

And this is from the LSP log panel:

:: [16:04:07.967]  -> LSP-copilot textDocument/didChange: {'textDocument': {'uri': 'file:///...', 'version': 42}, 'contentChanges': [{'range': {'start': {'line': 2860, 'character': 53}, 'end': {'line': 2860, 'character': 53}}, 'rangeLength': 0, 'text': 'c'}]}
:: [16:04:08.013] --> LSP-copilot getCompletions (6): <params with 147614 characters>
:: [16:04:08.027] --> LSP-copilot getCompletionsCycling (7): <params with 147614 characters>
:: [16:04:08.133] <-  LSP-copilot statusNotification: {'status': 'InProgress', 'message': ''}
:: [16:04:08.156] <-  LSP-copilot statusNotification: {'status': 'InProgress', 'message': ''}
:: [16:04:12.447] <-  LSP-copilot window/logMessage: {'type': 3, 'message': '[fetchCompletions] request.response: [https://copilot-proxy.githubusercontent.com/v1/engines/copilot-codex/completions] took 4288 ms'}
:: [16:04:12.920] <-  LSP-copilot window/logMessage: {'type': 3, 'message': '[streamChoices] solution 0 returned. finish reason: [Iteration Done]'}
:: [16:04:12.920] <-  LSP-copilot window/logMessage: {'type': 3, 'message': '[streamChoices] request done: headerRequestId: [] model deployment ID: []'}
:: [16:04:12.920] <-  LSP-copilot statusNotification: {'status': 'Normal', 'message': ''}
:: [16:04:12.920] <<< LSP-copilot (7) (duration: 4892ms): {'completions': [{'uuid': '4224f736-39f9-402e-b80e-027700892012', 'text': '\t\t\t\t\'title\'  => \'<span class="ab-icon dashicons dashicons-groups"></span>...', {'line': 2860, 'character': 54}, 'docVersion': 42, 'point': 105676, 'region': (105622, 105676)}]}

Verdict

So far it showed that it is neither better nor faster than Github Copilot: In the logfile above you can see that a completion took almost 5 seconds. But ollama-copilot works offline which is better than no copilot. And it works with only a few moving parts.

cll Now Works With Local Files And Improves Output Formatting

I’ve written about my cll tool before and it is still my go-to way of communicating with LLMs. See the Github repo. As a developer, having llms available in the Terminal is very helpful to me.

Write a file to disk

A lot of my prompts ask the LLM to create a file for me. This is often a fast starting point for working on something new. It makes me realize how much time it takes to start from zero and even if the LLM doesn’t get all details right on first try, it gives me a boost.

But with this also comes that I need to do a lot of copy-pasting from LLM output. So, if you already anticipate that you’ll receive a file, you can set the -f option and it will write the suggested file contents to disk:

cll -t please write me a wordpress php plugin that will log all insert and update calls to a custom post type Model: gpt-40-mini via OpenAI System prompt: When recommending file content it must be prepended with the proposed filename in the form: "File: filename.ext" > please write me a wordpress php plugin that will log all insert and update calls to a custom post type File: log-custom-post-type.php *"php <?php /**| * Plugin Name: Custom Post Type Logger * Description: Logs all insert and update calls to a specified custom post type. * Version: 1.0 * Author: Your Name */ [...]| // Hook into the save_post action. add _action( 'save_post', 'log_custom_post_type_changes', 10, 2 ); Instructions: 1. Replace 'your_custom_post_type' with the actual name of your custom post type. 2. Save the code into a file named log-custom-post-type.php. 3. Upload the file to your WordPress installation's wp-content/plugins directory. 4. Activate the plugin through the WordPress admin interface. 5. All insert and update actions for the specified custom post type will be logged to a file named custom-post-type-log. txt in the same di rectory as the plugin file. Adjust the logging method as needed for your environment or logging preferences. Writing 1248 bytes to file: log-custom-post-type.php
The quoted code is automatically written to a file.

Modify a file (or multiple)

Further conversation would update the file from above but you can also start with an existing file. I had previously modified cll to handle pipe input well but it can be very useful to just give it a local file using the -i parameter and if you combine it with -f it will automatically update the file as well.

cll -i log-custom-post-type.php -f modify this so that the insert statements are stored in a custom post type Model: gpt-40-mini via OpenAI System prompt: When recommending file content it must be prepended with the proposed filename in the form: "File: filename.ext" > modify this so that the insert statements are stored in a custom post type Local File: log-custom-post-type.php 1248 bytes: * Plugin Name: custom Post lype Logger. * Description: Logs all insert and update calls to a specified custom post type. * Version: 1.01 Add file content to the prompt? Ly/NJ: y File: 100-custom-post-type.php pho * Plugin Name: Custom Post Type Logger * Description: Logs all insert and update calls to a specified custom post type. * Version: 1.0| * Author: Your Name| */| [...]| add action init, register Log entry cor h Backing up existing Tile: 10g-custom-post-type.onp = Log-custo-post-type.ono.oak.1/24005124 Writing 2459 bytes to file: 10g-custom-post-type.php → log-custom-post-type.php.bak. 1724665121
The file is read and given to the LLM and then written to the file again.

Output Formatting

LLM output is often Markdown-like with headlines, bold text, inline code and code blocks. This now prints more nicely:

A screenshot of the CLL tool that shows formatted output for headlines, quoted code and bold text
It uses ANSI escape codes to get styled text in the Terminal.

These little additions keep cll useful for me. I know that it’s the typical engineer’s “I’ll roll my own” but like that it automatically falls back to Ollama locally if there is no network, has the nice output formatting, can work with files, and is always quickly available in the Terminal. You can use it, or get inspired for what you’d ask from a CLI LLM client. Checkout the Github repo at https://github.com/akirk/cll