Inferencer v1.6.1 with Distributed ServerAPI

Version 1.6.1 is now available for download on the Mac App Store. This update introduces support got multi-window sessions, distributed compute compatibility for OpenAI compatible API, code block copying and more.

Multi-Window support

In preparation for the upcoming batching feature, multiple windows support allows you to more easily work on different tasks simultaneously, side by side. Just press the Command + N keyboard shortcut to open a new window and try it out.

Distributed Compute with OpenAI Compatible API

Distributed compute is now supported with the OpenAI compatible API. To enable, start a server with OpenAI compatibility on the client. The models endpoint will now return a list of all the local models, remote models its connected to, and distributed compatible models. So just select the <distributed> version of the model and your client will handle the rest.

Please note that this is a preview feature, so if you run into any issues, please let us know.

P.S. If you find Inferencer useful, please consider leaving a review on the App store, it would be much appreciated.

More improvements

In the next update we’ll be bringing tool calling support, you can watch this video for a sneak preview, but more on that later.

As always, if you have any features or suggestions you’re more than welcome to add them to our public roadmap.

Thanks again for your support, and as a reminder, all AI processing is done offline, directly on your devices. No telemetry, no background "update" checks.