Distributed inference with P2P

Distribute computation by sharing and loadbalancing instances or sharding model weights.

LocalAI uses P2P technologies to enable distribution of work between peers. It is possible to share an instance with Federation and/or split the weights of a model across peers (only available with llama.cpp models). You can now share computational resources between your devices or your friends!

Network token

b3RwOgogIGRodDoKICAgIGludGVydmFsOiAxMjAKICAgIGtleTogQ2ZURlBraG92d0dQR285SjRacDVscGI4ME5UVUgyTlg4SHZ0TnpWaDFOUwogICAgbGVuZ3RoOiA0MwogIGNyeXB0bzoKICAgIGludGVydmFsOiA5MDAwCiAgICBrZXk6IEhMWkthczlySDE3QTJNV1NoRHpRclU1ak5jdjJDT3RIT1VBa2t1dno4a1YKICAgIGxlbmd0aDogNDMKcm9vbTogZTFkajM4dEFRTVZzNFVtNTRjTThjWlE3YThQN2YyZGd4NEltUUhXeld1YQpyZW5kZXp2b3VzOiBTQlp5VWlsVGJSM3NSY0taeWFxSVVLUG9XNWdXbU5sMUpaQWNKVEszWlB3Cm1kbnM6IE9LQjRhSkhOOGx1TmNRU3liblVramtvRmI1SmRGY1JTTUt3ajNwR3dRSGgKbWF4X21lc3NhZ2Vfc2l6ZTogMjA5NzE1MjAK

The network token can be used to either share the instance or join a federation or a worker network. Below you will find a few examples on how to start a new instance or a worker with the token and you will be able to see the available workers and federated nodes.

Federated Nodes:

You can start LocalAI in federated mode to share your instance, or start the federated server to balance requests between nodes of the federation.

To start a new instance to share:


                            # Start a new instance to share with --federated and a TOKEN

                            export TOKEN="b3RwOgogIGRodDoKICAgIGludGVydmFsOiAxMjAKICAgIGtleTogQ2ZURlBraG92d0dQR285SjRacDVscGI4ME5UVUgyTlg4SHZ0TnpWaDFOUwogICAgbGVuZ3RoOiA0MwogIGNyeXB0bzoKICAgIGludGVydmFsOiA5MDAwCiAgICBrZXk6IEhMWkthczlySDE3QTJNV1NoRHpRclU1ak5jdjJDT3RIT1VBa2t1dno4a1YKICAgIGxlbmd0aDogNDMKcm9vbTogZTFkajM4dEFRTVZzNFVtNTRjTThjWlE3YThQN2YyZGd4NEltUUhXeld1YQpyZW5kZXp2b3VzOiBTQlp5VWlsVGJSM3NSY0taeWFxSVVLUG9XNWdXbU5sMUpaQWNKVEszWlB3Cm1kbnM6IE9LQjRhSkhOOGx1TmNRU3liblVramtvRmI1SmRGY1JTTUt3ajNwR3dRSGgKbWF4X21lc3NhZ2Vfc2l6ZTogMjA5NzE1MjAK"

                            local-ai run --federated --p2p

Note: If you don't have a token do not specify it and use the generated one that you can find in this page.

To start a new federated load balancer:


                            export TOKEN="b3RwOgogIGRodDoKICAgIGludGVydmFsOiAxMjAKICAgIGtleTogQ2ZURlBraG92d0dQR285SjRacDVscGI4ME5UVUgyTlg4SHZ0TnpWaDFOUwogICAgbGVuZ3RoOiA0MwogIGNyeXB0bzoKICAgIGludGVydmFsOiA5MDAwCiAgICBrZXk6IEhMWkthczlySDE3QTJNV1NoRHpRclU1ak5jdjJDT3RIT1VBa2t1dno4a1YKICAgIGxlbmd0aDogNDMKcm9vbTogZTFkajM4dEFRTVZzNFVtNTRjTThjWlE3YThQN2YyZGd4NEltUUhXeld1YQpyZW5kZXp2b3VzOiBTQlp5VWlsVGJSM3NSY0taeWFxSVVLUG9XNWdXbU5sMUpaQWNKVEszWlB3Cm1kbnM6IE9LQjRhSkhOOGx1TmNRU3liblVramtvRmI1SmRGY1JTTUt3ajNwR3dRSGgKbWF4X21lc3NhZ2Vfc2l6ZTogMjA5NzE1MjAK"

                            local-ai federated

Note: Token is needed when starting the federated server.

For all the options available, please refer to the documentation.

To start a new federated instance:


                            docker run -ti --net host -e TOKEN="b3RwOgogIGRodDoKICAgIGludGVydmFsOiAxMjAKICAgIGtleTogQ2ZURlBraG92d0dQR285SjRacDVscGI4ME5UVUgyTlg4SHZ0TnpWaDFOUwogICAgbGVuZ3RoOiA0MwogIGNyeXB0bzoKICAgIGludGVydmFsOiA5MDAwCiAgICBrZXk6IEhMWkthczlySDE3QTJNV1NoRHpRclU1ak5jdjJDT3RIT1VBa2t1dno4a1YKICAgIGxlbmd0aDogNDMKcm9vbTogZTFkajM4dEFRTVZzNFVtNTRjTThjWlE3YThQN2YyZGd4NEltUUhXeld1YQpyZW5kZXp2b3VzOiBTQlp5VWlsVGJSM3NSY0taeWFxSVVLUG9XNWdXbU5sMUpaQWNKVEszWlB3Cm1kbnM6IE9LQjRhSkhOOGx1TmNRU3liblVramtvRmI1SmRGY1JTTUt3ajNwR3dRSGgKbWF4X21lc3NhZ2Vfc2l6ZTogMjA5NzE1MjAK" --name local-ai -p 8080:8080 localai/localai:latest-cpu run --federated --p2p

To start a new federated server with Docker (port to 9090):


                            docker run -ti --net host -e TOKEN="b3RwOgogIGRodDoKICAgIGludGVydmFsOiAxMjAKICAgIGtleTogQ2ZURlBraG92d0dQR285SjRacDVscGI4ME5UVUgyTlg4SHZ0TnpWaDFOUwogICAgbGVuZ3RoOiA0MwogIGNyeXB0bzoKICAgIGludGVydmFsOiA5MDAwCiAgICBrZXk6IEhMWkthczlySDE3QTJNV1NoRHpRclU1ak5jdjJDT3RIT1VBa2t1dno4a1YKICAgIGxlbmd0aDogNDMKcm9vbTogZTFkajM4dEFRTVZzNFVtNTRjTThjWlE3YThQN2YyZGd4NEltUUhXeld1YQpyZW5kZXp2b3VzOiBTQlp5VWlsVGJSM3NSY0taeWFxSVVLUG9XNWdXbU5sMUpaQWNKVEszWlB3Cm1kbnM6IE9LQjRhSkhOOGx1TmNRU3liblVramtvRmI1SmRGY1JTTUt3ajNwR3dRSGgKbWF4X21lc3NhZ2Vfc2l6ZTogMjA5NzE1MjAK" --name local-ai -p 9090:8080 localai/localai:latest-cpu federated

For all the options available and see what image to use, please refer to the Container images documentation and CLI parameters documentation.

Workers (llama.cpp):

You can start llama.cpp workers to distribute weights between the workers and offload part of the computation. To start a new worker, you can use the CLI or Docker.

Start a new llama.cpp P2P worker

CLI
Container images

To start a new worker, run the following command:


                            export TOKEN="b3RwOgogIGRodDoKICAgIGludGVydmFsOiAxMjAKICAgIGtleTogQ2ZURlBraG92d0dQR285SjRacDVscGI4ME5UVUgyTlg4SHZ0TnpWaDFOUwogICAgbGVuZ3RoOiA0MwogIGNyeXB0bzoKICAgIGludGVydmFsOiA5MDAwCiAgICBrZXk6IEhMWkthczlySDE3QTJNV1NoRHpRclU1ak5jdjJDT3RIT1VBa2t1dno4a1YKICAgIGxlbmd0aDogNDMKcm9vbTogZTFkajM4dEFRTVZzNFVtNTRjTThjWlE3YThQN2YyZGd4NEltUUhXeld1YQpyZW5kZXp2b3VzOiBTQlp5VWlsVGJSM3NSY0taeWFxSVVLUG9XNWdXbU5sMUpaQWNKVEszWlB3Cm1kbnM6IE9LQjRhSkhOOGx1TmNRU3liblVramtvRmI1SmRGY1JTTUt3ajNwR3dRSGgKbWF4X21lc3NhZ2Vfc2l6ZTogMjA5NzE1MjAK"

                            local-ai worker p2p-llama-cpp-rpc

For all the options available, please refer to the documentation.

To start a new worker with docker, run the following command:


                            docker run -ti --net host -e TOKEN="b3RwOgogIGRodDoKICAgIGludGVydmFsOiAxMjAKICAgIGtleTogQ2ZURlBraG92d0dQR285SjRacDVscGI4ME5UVUgyTlg4SHZ0TnpWaDFOUwogICAgbGVuZ3RoOiA0MwogIGNyeXB0bzoKICAgIGludGVydmFsOiA5MDAwCiAgICBrZXk6IEhMWkthczlySDE3QTJNV1NoRHpRclU1ak5jdjJDT3RIT1VBa2t1dno4a1YKICAgIGxlbmd0aDogNDMKcm9vbTogZTFkajM4dEFRTVZzNFVtNTRjTThjWlE3YThQN2YyZGd4NEltUUhXeld1YQpyZW5kZXp2b3VzOiBTQlp5VWlsVGJSM3NSY0taeWFxSVVLUG9XNWdXbU5sMUpaQWNKVEszWlB3Cm1kbnM6IE9LQjRhSkhOOGx1TmNRU3liblVramtvRmI1SmRGY1JTTUt3ajNwR3dRSGgKbWF4X21lc3NhZ2Vfc2l6ZTogMjA5NzE1MjAK" --name local-ai -p 8080:8080 localai/localai:latest-cpu worker p2p-llama-cpp-rpc

For all the options available and see what image to use, please refer to the Container images documentation and CLI parameters documentation.