01-27-2025: We updated this blog post with new information about recent LLMs, a simplified setup method, and specific instructions on using Deepseek-R1.
Congratulations, you’ve decided to enter the chaotic, thrilling, and occasionally nonsensical world of LLM validation! Maybe your CTO asked you to “just spin up a model real quick.” Maybe you’re on a quest to prove that Deepseek-R1 is the next big thing—or at least better than your company's last AI-powered attempts at collaborating on an open source AI or integrating its outputs into your own apps.
Usually, even validating this kind of work sends you down one of two rabbit-holes: First, you set all this up on your local machine, which is fragile, unrepeatable, and decidedly not GPU-accelerated. Second, you over-complicate and end up with a massive—and expensive—deployment before you've gotten to hello, world.
Today, we're going to show you a just-right setup—including ngrok, Deepseek, and Ollama—to help you test, collaborate on, and validate AI models like Deepseek-R1 without exposing them to the entire internet. Let’s dive in before someone suggests retraining GPT-2 to save a few bucks.
The obvious first choice is to develop your LLM locally, as you do with other development work, committing changes and pushing them to a Git provider like GitHub.
Rodrigo Rocco, for example, pleasantly surprised us when he showcased on Twitter/X how he’s running AI models locally and making results available externally through an ngrok tunnel. That said, there are limitations to the local-first model.
First, the most powerful LLMs require 16GB, 32GB, or even more RAM, plus a discrete GPU for churning through LLM-specific workloads. Your local workstation might not be that powerful now, and improvements aren’t in the budget. Or, even if you have a newer system, you don’t feel like taxing it, potentially slowing down other work, while you fine-tune a large language model (LLM) locally.
There are other limitations around collaboration and eventual hosting. A local development workflow doesn’t allow others to collaborate with you easily unless you expose your workstation to the public internet and leave it on 24/7. The same goes for the eventual transition into an API—better to build on a platform you won’t have to migrate off of inevitably.
When looking specifically at remote compute for LLM development, you have two choices:
Hosted options are great for fast-moving startups that need to launch and train LLMs with the absolute minimum infrastructure, but you need something that’s both owned by your organization and persistent for continuous development.
Self-hosting is the best choice for this situation, but comes with new technical challenges.
Securely connecting your local workstation to a remote service requires you to deal with proxies, port forwarding, firewalls, and so on. If you were to bring an LLM service into production using the normal route—making a formal request with your DevOps peers—it might take weeks of code and coordination to get it running for security and compliance.
That said, companies like Softwrd.ai and Factors.ai are already using ngrok to connect to remote CPU/GPU compute, getting new AI-based APIs to market fast, which means you’re not the first to venture into this proof of concept using a powerful solution: ngrok.
You’ve gone through all the technical requirements of building this proof of concept AI workflow—time for everyone’s favorite part of building new tech: deciding on your stack.
Based on the CTO’s goals and the roadblocks you’ve already found, a viable option consists of:
This stack s by no means the only way of connecting your local development workflows to remote compute power. We’ve chosen it here due to its simplicity to get started:
There are some disadvantages to this approach, too:
Head on over to the Google Cloud Console and Create a VM.
Under Machine configuration, select GPUs and pick the GPU type that works for your budget and needs. For the machine type, pick a high memory instance, like n1-highmem-2. Down in the Boot disk section, click Switch Image to get an optimized operating system like Deep Learning on Linux, and up the size of the disk to 100 GB to be on the safe side. Down in the Firewall section, click Allow HTTPS traffic—ngrok will use that later to make your LLM accessible from anywhere.
That should be the fundamentals you need to launch remote AI compute—at about $0.35 per hour.
Give your instance some time to fire up.
When it’s ready, SSH into it with your preferred method.
The first time you log in, your VM will prompt if you want to install Nvidia drivers—hit y and Enter, as the reason you’re paying extra for VM is access to GPU compute.
Once that’s done, you can run nvidia-smi to verify that your GPU acceleration works as expected.
nvidia-smi
Fri Jan 19 18:15:27 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 70C P0 33W / 70W | 2MiB / 15360MiB | 9% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+The fastest way to get Ollama working is their Bash one-liner, but you can also opt for a manual installation or Docker.
curl -fsSL https://ollama.com/install.sh | shWith Ollama installed, pull and run deepseek-r1.
ollama pull deepseek-r1You now have Deepseek-R1 available on Ollama, which is already running a server on port 11434.
That's great, but since said server is running on the remote VM, you need a way to connect to chats and completions without ssh-ing in or opening port 11434 for attackers and bots to discover and abuse.
To download the ngrok agent, head on over to the downloads page or quickstart doc for multiple options compatible with any Linux VM, Debian and beyond. Make sure you grab your Authtoken from your ngrok dashboard and connect your account.
While you have the ngrok dashboard open, reserve a new ngrok domain—this is your only method of having a consistent tunnel URL for all your future LLM training operations using this stack, either using a subdomain of ngrok.app or a custom domain you own.
The following command will create an ngrok endpoint, which routes all traffic from {YOUR_LLM_DOMAIN}.ngrok.app to port 11434 on your GPU-powered VM.
Per the Ollama docs, the --host-header flag ensures Ollama receives the expected Host header.
The ngrok agent immediately starts forwarding any API requests on your reserved domain to Ollama and your new Deepseek-R1 model.
ngrok http 11434 --url={YOUR_LLM_DOMAIN} --host-header="localhost:11434"You can now send API requests directly to your GPU-powered Deepseek-R1 model.
curl https://{YOUR_LLM_DOMAIN}/api/generate -d '{
"model": "deepseek-r1",
"prompt":"Why is the sky blue?",
"stream": false,
}'Depending on the horsepower of the VM you chose, you'll see a similar result soon enough.
{
"model": "deepseek-r1",
"created_at": "2025-01-27T21:46:52.004359039Z",
"response": "\u003cthink\u003e\n\n\u003c/think\u003e\n\nThe sky appears blue due to a phenomenon called Rayleigh scattering. When sunlight reaches Earth's atmosphere, it interacts with molecules and small particles in the air. Blue light scatters more frequently than other colors because it travels shorter wavelengths and interacts more with the atmosphere. This scattering effect is why we see the sky as blue during the day.\n\nAdditionally, during sunrise or sunset, the sky often appears red or orange. This is due to a phenomenon called Rayleigh scattering combined with the way our eyes perceive light. When sunlight passes through layers of the atmosphere, the blue light is scattered away, leaving the longer wavelengths (red and orange) to dominate the sky's appearance.",
"done": true,
"done_reason": "stop",
"context": [
151644, 10234, 374, 279, 12884, 6303, 30, 151645, 151648, 271, 151649, 271,
785, 12884, 7952, 6303, 4152, 311, 264, 24844, 2598, 13255, 62969, 71816,
13, 3197, 39020, 24491, 9237, 594, 16566, 11, 432, 83161, 448, 34615, 323,
2613, 18730, 304, 279, 3720, 13, 8697, 3100, 1136, 10175, 803, 13814, 1091,
1008, 7987, 1576, 432, 34192, 23327, 92859, 323, 83161, 803, 448, 279,
16566, 13, 1096, 71816, 2456, 374, 3170, 582, 1490, 279, 12884, 438, 6303,
2337, 279, 1899, 382, 49574, 11, 2337, 63819, 476, 42984, 11, 279, 12884,
3545, 7952, 2518, 476, 18575, 13, 1096, 374, 4152, 311, 264, 24844, 2598,
13255, 62969, 71816, 10856, 448, 279, 1616, 1039, 6414, 44393, 3100, 13,
3197, 39020, 16211, 1526, 13617, 315, 279, 16566, 11, 279, 6303, 3100, 374,
36967, 3123, 11, 9380, 279, 5021, 92859, 320, 1151, 323, 18575, 8, 311,
40736, 279, 12884, 594, 11094, 13
],
"total_duration": 43516935923,
"load_duration": 7419493865,
"prompt_eval_count": 9,
"prompt_eval_duration": 1455000000,
"eval_count": 138,
"eval_duration": 34639000000
}Now that you have an LLM running, you absolutely want to limit who can access it.
Through our Traffic Policy engine, ngrok offers many possibilities:
For the sake of simplicity, let's implement the Basic Auth action using Traffic Policy.
Close down ngrok on your VM and create a new file called policy.yaml with the following.
---
on_http_request:
- actions:
- type: basic-auth
config:
credentials:
- user1:password1
- user2:password2These lines of YAML implement the Basic Auth action, which checks for Base64-encoded credentials with every request. All requests without these credentials receive a 401 Unauthorized response directly from ngrok's network without even touching your VM.
You can also add up to 10 total credentials for different people on your team.
Start up the ngrok agent again with your new Traffic Policy file.
ngrok http 11434 --url={YOUR_LLM_DOMAIN} --host-header="localhost:11434" --traffic-policy-file=policy.yamlWith your next request to Deepseek-R1, use curl's -u flag, which Base64-encodes the credentials for you.
curl -u user1:password1 \
https://{YOUR_LLM_DOMAIN}/api/generate -d '{
"model": "deepseek-r1",
"prompt":"Why is ngrok so great?",
"stream": false,
}'To be honest, we're not quite sure at this point. Things are moving fast.
What matters most is that you now have an effective testbed for open source LLMs like Deepseek-R1, whether you're using it to collaborate with your coworkers, trying to validate whether you could integrate it into an existing app, or just trying to compare this new model's results to LLMs you've used before.
As you experiment with Deepseek or other LLMs and eventually want to make your deployment more secure, I have a few recommendations:
We'd love to know what you loved learning from this project by pinging us on X (aka Twitter) @ngrokhq or LinkedIn. Have other questions about using ngrok in production? Join us on our monthly Office Hours livestreams for demos and Q&A from our DevRel and Product teams.