But it's 2026, and the game has changed. You no longer need to be a Linux wizard to have a private, powerful AI in your pocket. In this guide, I'm covering the Top 4 ways you can run LLMs locally on your Android or iOS device—completely for free, with unlimited requests, and zero data leaving your phone.

1. Google AI Edge Gallery: The "One-Click" Starter

https://github.com/google-ai-edge/gallery

Google's official showcase is the perfect starting point. It's the easiest way to see what your phone is actually capable of. It's optimized for mobile NPUs, meaning it's fast and efficient for tasks like tone refinement, audio transcription, and image understanding.

The latest version now supports Gemma 4 and Gemma 3, models specifically "distilled" to run on mobile hardware without eating up your RAM.

The "Skills" Update: One of the coolest additions is support for SKILLS. You can now extend what the AI can do by pulling from a vast library at skills.sh. My personal favorite is the Humanizer skill—it's great for making AI-generated text sound more, well, human.

Demo

https://youtube.com/shorts/1H6z2-K7gvw

Models Available

Bundling your own model

If you're feeling adventurous, you can even bundle your own models by following the LiteRT community guide. It walks you through converting models from Pytorch or other formats.

2. PocketPal AI

PocketPal AI is the one I keep coming back to. It's built on llama.cpp (the industry standard), but the interface is actually designed for a human using a phone, not a developer staring at a terminal.

The 'Pals' feature is the standout here. Instead of one messy chat thread, you can set up different "Pals" with their own system prompts. I have one for coding and another for creative writing. It also handles the latest architectures like Qwen 3.5 and LFM 2.5, which are incredibly fast on mobile.

Why it works for me:

• Easy Model Setup: You can download GGUF models directly within the app. For a smooth experience, I recommend any GGUF model (try Unsloth's uploads) with Q4_K_M quantization.

• 100% Private: Once the model is on your phone, you can go into Airplane Mode and the AI won't even notice.

• Live Stats: It shows you tokens per second in real-time. It's satisfying to see your phone's hardware in action.

3. AnythingLLM: The Researcher's Workspace

If you need your AI to actually do something with your files, AnythingLLM is a different beast. It's less of a chatbot and more of a portable workspace.

The killer feature is On-Device RAG. You can feed it a PDF or text file sitting on your phone, and the AI will answer questions based only on that document. I've used this to summarize 50-page technical docs during flights with zero internet.

What sets it apart:

• Chat with your docs: Local indexing means your data never touches a server.

• Tools & Agents: It supports web scraping, calendar editing, and the Model Context Protocol (MCP), so it's slowly becoming a true mobile agent.

• The "Infinite Power" Trick: If your phone is struggling, you can connect it to a massive 70B model running on your home PC via API and use your phone as a remote window into that power.

4. Termux + Ollama: The Power User Shortcut

This is for my Android users who miss the command line. We're using Termux to run Ollama directly on the device. It's the closest you'll get to a desktop experience on a mobile screen.

The Quick Setup:

1. Install Termux (use the version from F-Droid, not the Play Store).

2. Run pkg install ollama.

3. Type ollama run qwen3.5:0.8b.

That's it. Ollama will auto-download the model and drop you into a CLI chat. You can run pretty much anything from the Ollama Library as long as your RAM can handle it. It's the fastest way to test new models the second they drop.

Summary: Which one should you pick?

The era of "Local-First" AI

The era of relying on expensive subscriptions and cloud-tracking for AI is ending. Whether you want a simple one-click app or a full terminal setup, you can now carry a "God-tier" brain in your pocket for $0.

Read about more on Local AI and Self-Hosting on nkaushik.in.

Happy (Local) Chatting!

But saving and viewing these memories has started to become costly, and with every tech giant racing to feed our data into AI models., these private memories of ours are also being used to train these new AI models.

I wanted to figure out how could I make these memories of mine more accessible, and actually be able to see them again with ease.

Earlier I used to manually copy and backup my photos to an external backup storage but recently I had my phone die on me which meant I lost all my recent photographs, so It was really important if I could make this backup automatic.

The 2TB plan on Google Photos is going to set you back 100\(/yr, while the same on iCloud will cost you 132\)/yr. But lets be real, how often do we really access these photos, so while having the ability to access them is important, you could optimize how you pay for it all.

The Best Google Photos Alternatives in 2026

There are a couple of Open Source options that I came across like, Ente Photos, Immich etc.

I went ahead with Immich, because I liked the interface, as well as while searching for the options I found a lot of positive sentiments about Immich in the reddit community.

Self-Host Immich

https://immich.app/

Self-hosted photo and video management solution

Features of Immich

Immich has pretty much all the feature you can think of

• Face Detection

• Searching based on text/tags

• Location Mapping (one of my favorites ❤️‍🔥)

• and a lot more…

Setting up Immich is quite straightforward, we can set it up using multiple strategies, in this guide I’ll be using docker, as that allows us to fresh-start in case we mess anything up.

Getting things ready

Docker & Compose

Follow the steps on https://docs.docker.com/get-started/get-docker/ to install docker and docker compose.

Because we are using docker, the setup is identical across all operating systems..

Setting things up

The Immich documentation is an excellent starting point, offering a straightforward, step-by-step guide.

Follow the steps here: https://docs.immich.app/install/docker-compose

:warning:

While you can have Immich store the photos library on either SSD or HDD, make sure you setup Immich and most importantly the database Immich uses on a SSD, this ensures everything stays snappy.

BUT If you do install Immich database on a HDD make sure to set the environment variable DB_STORAGE_TYPE with the appropriate value https://docs.immich.app/install/environment-variables#database

Once you have completed all the steps you will have your personal Google Photos alternative i.e., Immich running on http://<machine-ip-address>:2283 / http://localhost:2283

Adding your existing albums

Mounting the directories

While Immich is meant for doing backup from your devices, It doesn’t mean you need to upload your photos manually for loading them into Immich. If you already have your photos and memories in a folder you can point them into Immich and it would load them for you.

Here’s an example of the docker-compose.yml file that I use:

services:
  immich_server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    ......
    volumes:
      .......
      # Modify below to change external albums path
      - /media/PhotosStorage:/mnt/media/og-photo-albums:ro # Change this to your media storage location
      ######
    .....
    restart: always

In the above compose file I use :ro to make my external albums location mount in read-only mode, You can change it to :rw to have it in read-write mode which will allow Immich to also delete and edit files.

Once you have added these folders into your docker-compose file, you need to restart your Immich instance,

You can do so by running the below command in the folder you have your Immich’s docker-compose file.

docker compose up -d

After restart you can monitor the logs by running the command docker compose logs in the same folder, to see if the Immich service has restarted successfully or not after adding your external photo albums.

Updating Immich to read external Albums

Now to make sure Immich loads these photos and show it in the UI. You need to configure it in the Admin setting.

Head over to your Immich instance, then from the Settings > Administration > External Libraries

Click on Create Library

On newly created External Library Click on + Add button to add your external albums

The path you add here is from your docker compose file which you previously added.

Example:

- /media/PhotosStorage:/mnt/media/og-photo-albums

Insert the path value you see on the right side of the :  i.e, in this case it would be /mnt/media/og-photo-albums

Triggering Jobs

Once you have added your external album, you can hit Scan to have Immich read all your photos and videos.

You can also manually trigger the jobs to generate thumbnails, doing duplicate detection, face detection etc by going to the Job Queues page

When you run it for the first time these jobs might take time depending upon the size and number of file in your folder, so be patient and let it run through.

You can check the status of what jobs are running and how much work is pending on the same page.

Once everything has finished, your external album is ready to use 🎉

Making it Public and Secure

To truly have Immich as a backup option, it is important that we can access it outside our home and make it public so that we can also do backups.

There are 2 ways to go about it:

1. Cloudflare Tunnel

2. Tailscale

Both of these methods have their own benefits and downsides, let me quickly cover those;

Cloudflare

• You must own a domain

• 100MB Limits on file uploads under free plan (problem for videos)

• Anyone can access without creating Cloudflare account

Tailscale

• No domain required

• VPN required to be connected always

• Need to invite users to Tailscale to access Immich

• No file upload limits

Depending upon which one you prefer you can follow either of these setup:

Cloudflare setup

1. Create a Cloudflare tunnel https://developers.cloudflare.com/cloudflare-one/networks/connectors/cloudflare-tunnel/get-started/

2. In the “Publish an application” step you would publish Immich which is running on localhost:2283

3. Once your tunnel is up, you can now access your Immich over the subdomain you created. eg., immich.domain.com

4. Open the above URL to verify if your Immich is running and is accessible and you land on the login page.

If you are able to access Immich, you can now use this URL in the Immich app.

Next, We can now also make this a bit more secure, such that not everyone can access even the login page of Immich, this would prevent unauthorized access to your Immich instance.

To do that, we will setup “Access Control”

Head over to https://developers.cloudflare.com/cloudflare-one/access-controls/applications/http-apps/self-hosted-public-app/ and setup a new application, while setting up the policies you can even control who can access this application of yours and filter based on the emails. When you setup the application now when you access the URL it would land on this type of login page.

Anyone can enter their email and request the code to get access, you can limit the emails that can do so, essentially blocking anyone else from even getting access.

Now, once we enable access control, it would prevent the Immich app from opening the URL.

To solve for it, we would add another “Policy” to the app called “Service Auth” which would be used in Immich app.

Follow the step on https://developers.cloudflare.com/cloudflare-one/access-controls/service-credentials/service-tokens/ and create a new Service Token.

You should now have two values, keep them safe and save it securely.

CF-Access-Client-Id: <CLIENT_ID>

CF-Access-Client-Secret: <CLIENT_SECRET>

Now, go back to your “Application” you created in the previous step and “Policies” tab.

Click “Select existing policies” and pick the newly created “Service Auth”.

Now, your Immich app will have two ways in which users can “access” it 1. Using the 1-time code by providing email, 2. By using the secret headers “CF-Access-Client-Id” and “CF-Access-Client-Secret” (which would be primarily for Immich app)

Now, To configure the Immich app

• Go to settings

• Advanced

• Custom proxy headers

• Add the values you saved previously

Now it should look like something below.

You can now add the URL in your “Networking” section of the Immich app settings.

You should see a green check-mark :white_check_mark: next to it if the app is able to access it.

Congrats your Immich app is now ready. 🎉

Tailscale setup

Go to https://login.tailscale.com/welcome and create a new account

Then “Add device” and install tailscale on your system.

Now your system should show up in the “Machines” tab.

Now go to “Services” tab and “Define a service”

Enter the name of the service and use port as ‘443’

It should look something like this:

After defining the service Tailscale should give you a command to run.

It might look like

tailscale serve --service=svc:immich --https=443 127.0.0.1:2283

run it in terminal on your system, and then accept the service on the Services tab in tailscale.

Once you finish this you should see “**🟢**Online” next to the service name.

Now connect to tailscale VPN using the client and try opening the URL you see next to the service.

You should now be able to access the Immich app over tailscale, remember this URL will only work if you are connected over Tailscale VPN.

You can also add this URL in the Immich app under the “Networking” section in Immich setting.

Once you see the green check-mark ✅ next to your tailscale URL in Immich app, you are done!

Your Immich app is now remotely accessible.🎉

Ending thoughts

Running your own Google Photos alternative like Immich lets you take back control of your memories — cutting recurring cloud costs, improving privacy, and automating backups so you don’t lose recent photos again.

It does require some upfront effort (hosting, storage planning, and occasional maintenance), but the tradeoffs are worth it if you value long-term accessibility and ownership of your data.

Start small: test with a subset of your library, automate uploads from your phone, and keep an offsite or secondary backup for redundancy.

With the active open-source community and clear docs, self-hosting is a practical, cost-effective option for anyone ready to invest a little time for greater control.

Whats Next?

I am writing about all the various service I have been self-hosting.

Read my other posts on Self-Hosting on:

https://nkaushik.in/writing/series/homelab/

https://nkaushik.in/writing/getting-started-with-self-hosting-in-2026

https://nkaushik.in/writing/adguard-home-for-families-how-to-stop-ads-and-trackers-without-installing-extensions/

Start your self-hosting journey in 2026 with affordable hardware, stable software like Debian, and containerization via Docker.

This guide is for anyone who want to take control of their data, starting with family photos and expanding to network-wide services. No advanced Linux knowledge is required.

Introduction

Do you also have 100s of GB of Photos and Videos of you and your family but you rarely see them?

That bothered me too. Finding a solution started my self-hosting journey.

While solutions like Google Photos existed the idea of having to upload all that data and then having to pay a subscription price for something that will not be accessed that frequently did not feel right.

I wanted something which could just reduce the friction of accessing my library and that's when I came across this great project known as Immich🔗.

What started as running a single self-hosted service, soon exploded, and now I run 30+ services self-hosted and accessible even publicly.

In this blog I'll go over how you can start your Self-Hosting Journey and How to choose the "right hardware" for your needs.

Hardware Overview

I started off by running Immich in a docker container on my Legion 5i Laptop with 16GB RAM, i7-14650HX, NVIDIA 4060 GPU, this allowed me to test things out first.

Once I was able to setup everything and have it run and felt it would work for my use-case, I started looking into hardware options, as running a big clunky laptop 24x7 was NOT an effective solution.

I debated buying a Raspberry Pi 5, but the price in India was high and combining it with its various HATs increased significantly and it became less of an option for me. Another limiting factor of Pi 5 was its future scalability as I was worried the Pi's hardware limitations would eventually prevent me from expanding into more complex services.

Ultimately, I settled and bought an old mini PC. I bought a refurbished HP Elitedesk 800 G2 mini for ₹11,500.

The configuration I currently have for my mini PC are:

CPU: i5-6500T

RAM: 8GB DDR4

Storage: 256 GB NVME SSD, 256GB Internal Sata SSD

External Storage: 500GB HDD connected over USB

Networking: 1Gbps Ethernet

I have my server connected to my router via an Cat 5e Ethernet cable.

How to choose the Hardware?

If you are just getting into self-hosting, then I would recommend first asking yourself these 2 questions:

1. Why do you want to do it? What services do you plan on self-hosting?

2. Who are you self-hosting for? How many users will this system support?

If the answer for 1st is only couple of services and the answer to 2nd question is < 5 then you don't need to necessarily invest big upfront and rather I would recommend utilizing any old PC/Laptop you may already have.

With that said, If you do want to buy a new hardware, I would recommend NOT buying a SBC like Raspberry Pi, simply because getting a Mini PC or an actual full size system will give you much more headroom should you choose to expand your services (which you will, speaking from experience 😅)

Go for at-least 8 or 16GB RAM and at-least 256GB of internal NVME SSD. While I use a 6th Gen i5-6500T, which remains an excellent budget entry point in 2026 due to its low power and cost, you might also consider 8th Gen+ Intel CPUs for official Windows 11 support and better QuickSync, or 10th/12th Gen for modern AV1 decoding/encoding. The key is that you can start with almost anything and scale up later.

You can treat the internal SSD as premium storage and use it only for the primary system and then have all your data stored on hard-disk which would be much more economical and would simply work for pretty much all the use-cases.

Software / OS

To effectively utilize my machine as a server, I installed Debian 12 (now upgraded to Debian 13) for its stability, something crucial for a system I plan on running 24x7.

As I planned on using terminal primarily to access my system, I installed Xfce4 as my desktop environment, to keep the system resource usage low and prioritize them for the services I would be running.

I then installed Docker & Docker Compose as I would be running all the services in containers, this would allow me to keep things clean on my system, also have all my services isolated and try out different services and restart from scratch if I mess things up.

I also set up SSH server to ensure I can access my machine via terminal and xrdp to be able to RDP into my system when needed.

In another blog I'll cover in detail all the software/packages I use and how i have them set up.

How to choose the OS?

If you are new to Linux, look for a distro which would be much closer in terms of the UI you are familiar with. OR if nothing use Debian, it is stable, it ships updates on a regular manner and you can just run it with install-and-forget mindset.

Regardless of what distro you choose you can always change the UI (desktop environment) later on. I started with Xfce but then also install KDE Plasma and now I can choose between the two when I log into the system directly.

Network Diagram

This is what my current network architecture looks like.

My server currently is connected directly to the Wifi Router over Ethernet and I tend to keep the WIFI off on it, and I have also assigned it a fixed IP in my Router settings in-case I plug it out from Ethernet and connect over WIFI.

The fixed IP is crucial, not just for ad-blocking, but to ensure you can always find your server via SSH or access your hosted services without the address changing.

You can read more about setting up ad-blocking here: https://nkaushik.in/writing/adguard-home-for-families-how-to-stop-ads-and-trackers-without-installing-extensions

Backups: Protect your Data

The most important rule of self-hosting is: RAID is not a backup.

Since your family photos are likely the most precious data you'll host, a robust backup strategy is critical. I recommend the 3-2-1 strategy:

• 3 copies of your data (Original + 2 backups).

• 2 different media types (e.g., Internal SSD and External HDD).

• 1 copy offsite (Cloud storage like Backblaze B2 or AWS S3 using tools like Restic for automated, encrypted backups).

We often overlook that with self-hosting, we are responsible for both reliability and security

What's Next?

Once you set these 3 main pillars i.e., Hardware, Software & Networking, you are ready to start your self-hosted journey.

This blog is just the beginning. In my Homelab Series, I'll be covering a lot more services in detail:

• Media: Setting up Jellyfin or Plex for your movie collection.

• Photos: A deep dive into Immich for your family memories.

• Monitoring: Using Uptime Kuma and Ntfy for real-time status and notifications.

• Remote Access: Safely accessing your services from anywhere via Cloudflare Tunnels or Tailscale and setting up Zero Trust

Update:

Checkout the other Self-Hosting posts:

• Immich Guide http://nkaushik.in/writing/replace-google-photos-self-host-immich-in-2026

• AdGuard Home https://nkaushik.in/writing/adguard-home-for-families-how-to-stop-ads-and-trackers-without-installing-extensions

Now go ahead and start hosting!

Although you might use apps like Firefox or extensions such as Adblock, they must be installed on every device, which is often impractical.

I faced a similar issue because my family, like many others, isn't tech-savvy. I needed a solution that could be implemented to shield them from ads and malware without requiring any changes on their part.

Enter, AdGuardHome

https://github.com/AdguardTeam/AdGuardHome is a wonderful thing. I have found it extremely easy to Install, Manage and Configure.

In, this blog I would cover how you can also Install AdGuard.

Installation

Head over to https://github.com/AdguardTeam/AdGuardHome?tab=readme-ov-file#automated-install-linux-and-mac and would notice the installation of AdGuard is a simple as running a command.

To install with curl run the following command:

curl -s -S -L https://raw.githubusercontent.com/AdguardTeam/AdGuardHome/master/scripts/install.sh | sh -s -- -v

To install with wget run the following command:

wget --no-verbose -O - https://raw.githubusercontent.com/AdguardTeam/AdGuardHome/master/scripts/install.sh | sh -s -- -v

This will install the AdGuard Service which would acts as your own DNS Server.

What is DNS?

DNS stands for Domain Name System, In very simple terms: You see it is like the internet's phone book, every website you visit actually has something known as IP Address, eg., 66.11.184.216, which as you can imagine is very hard to remember, so you add an entry into the phone-book which says when you look for example.com , It makes a call to 66.11.184.216.

DNS is essential because it allows us to use friendly names instead of numbers.

Setting up AdGuard Home

https://adguard-dns.io/kb/adguard-home/getting-started/

After installation, a web interface should start on port 3000, allowing you to configure AdGuard. Just open the URL http://localhost:3000 to set it up.

Step 1. Click Get Started

Step 2:

Here you can configure on which port AdGuard will run its UI for you to configure it after installation, by default it uses port 80, I highly recommend changing it.

Next, you'll have the option to change the port on which your DNS server runs after installation. By default, it uses port 53, and I recommend keeping it as is.

Step 3:

Setup your admin credentials to manage AdGuard

continue through the steps to finish setting up.

After completing all the steps, you can access the admin interface at the port you configured in step 2.

When you arrive at the admin page, it may appear empty because you still need to configure your devices to use AdGuard.

You can add the filters by going to Filters > DNS blocklists from the top nav-bar.

Here's a list of all the block-lists I have enabled.

Using AdGuard with your devices

The most optimal way to install AdGuard is to configure it at the Router level.

Head over to the Setup Guide page on your AdGuard admin interface and you can find the details and steps to set it up on different devices.

Before setting it up on your router, I recommend configuring it only on your system first. This ensures you won't lose internet access if something goes wrong, and make adjustments as needed..

Example for windows systems, you can follow these steps:

Once, you have verified it on a single device, and you can see traffic flowing into your AdGuard instance i.e,. you see DNS Queries showing up on your AdGuard interface, You can go ahead and configure it at the router level.

In your router's admin panel, locate the DHCP settings and enter the IP address where your AdGuard is running in the DNS field. You can find this IP on the Setup Guide page; it will resemble something like 192.168.xx.xxx.

Once you've configured it at the router level, all devices connected to your home network will have AdGuard enabled.

Bonus

In addition to blocking ads and trackers, AdGuard offers the added benefit of enabling content blocking across your entire home network.

It offers an easy-to-use method to block services you want to restrict, such as adult content and gambling.

You can take it a step further by blocking services for specific time periods. For example, you can restrict access to YouTube for your kids outside designated hours, ensuring effective parental control.

Conclusion

AdGuard Home gives you a simple, network-wide way to reclaim privacy, reduce intrusive ads, and add a layer of protection against malicious domains — all without installing browser extensions on every device. Once installed on a small always-on machine (Raspberry Pi, home server, or VPS), it works silently for every device that uses your network DNS, making it ideal for non-technical family members.

To finish your setup: point devices or your router to AdGuard Home as the DNS, enable the appropriate blocklists and filters, whitelist sites you trust, and secure the web UI with a strong password (or VPN). Regularly update filter lists and back up your configuration. Monitor logs briefly after deployment to catch any false positives and adjust rules as needed.

AdGuard Home is low-maintenance but powerful — a practical step toward a cleaner, safer browsing experience for everyone on your network.

Why? Because it opened the door to the world of self-hosting. Initially, my goal was simply to organize my photo library, which had been tucked away on my hard disk drive. However, I ended up running over 35 services in Docker containers!

[Here's a preview of all the services I currently run on my server.]

While I run a plethora of services, there are a couple of services that are top-tier for me, namely:

• Adguard: For Adblocking and custom DNS

• Immich: The Google Photos alternative

• Jellyfin: For all the Linux ISOs 🙂

• Outline Wiki: For internal notes, documentation and wikis

These are some of the services I use frequently, while others quietly work behind the scenes, acting as the unsung heroes that keep everything running smoothly.

To Begin

In this multi-part series, I will cover how I configured each of these services, along with the best practices I used to make them publicly or remotely accessible and secure.

You can find all the details in the series:

https://nkaushik.in/writing/series/homelab

In this follow-up I address the biggest obstacle to running the device 24/7: overheating and how I transformed it into a fully functional personal local AI solution.

Fixing Heating Problem

One of the main challenges I faced was running it 24/7 without overheating the device.

Fortunately, during a brainstorming session with Gemini, I learned about Advanced Charging Control (ACC), a Magisk Module. https://github.com/VR-25/acc

Having already rooted the device for tinkering purposes, I decided to give it a try. Some of the latest phones offer the option to skip charging and run directly on AC power, but since my device was older and lacked this feature, the Magisk module gave me hope.

The process was pretty straightforward to install the module, and once done, I had to reboot the device.

After rebooting, my device could now automatically turn charging on and off at specific battery levels. I kept the default settings, which meant it would stop charging at 75% and start again at 70%. This module prevented constant charging, helping reduce the heat generated from being plugged in continuously.

Llama.cpp Server

Another optimization I implemented involved running the models more efficiently. Previously, I relied on a third-party app to connect to the llama.cpp server on the device, but these apps were often unreliable and glitchy. I discovered that running the llama-server also launches a web UI, accessible at http://localhost:8080, unless explicitly disabled. This allowed me to bypass the faulty third-party apps and directly access the models through the URL.

I faced another challenge: how to switch models according to my needs. I had heard of llama-swap, but since I was running these commands on my Android device, compiling another binary was something I wanted to avoid. Then I found a blog on Hugging Face about model management in llama.cpp, which mentioned that a recent update to llama.cpp had introduced a solution for this issue.

You can now run llama-server --models-dir ./models, allowing users to choose the desired model. In the web UI, you can easily select the model from a drop-down menu.

I simply created an alias that worked for my case and ended up with

llama-server --models-dir ./storage/downloads/models --models-max 1 -c 8192 --sleep-idle-seconds 120

Now, when I open the server running on port 127.0.0.1:8080, I see something like this:

I conducted a simple test on several models by giving each one the identical prompt. After submitting the prompt, I recorded the tokens per second (t/s) performance for each model.

As expected, the 0.6B parameter model was the fastest, while the 1.7B model was the slowest. Both Qwen3 models required a considerable amount of time. The Qwen3 4B model was particularly slow, achieving only about 2 tokens per second, making it impractical for most uses.

Demo

Here's an example of the model in action with a sample prompt I provided:

https://youtu.be/k0-X5YokIJ4

Closing thoughts

In short, using the Advanced Charging Control Magisk module let me run local AI workloads on an older Android continuously without the device constantly charging and overheating. By having the phone stop charging around 75% and resume at 70%, the battery no longer acted as a heat sink while plugged in, which noticeably reduced sustained temperatures and made 24/7 operation practical on hardware that otherwise would have run too hot.

Key takeaways:

• Rooting and ACC worked well for my device but carries risks (voided warranty, potential bricking); proceed only if you’re comfortable with those trade-offs.

• Use conservative charge thresholds and monitor temperatures after changes; what’s safe depends on your phone’s age and thermal design.

• Combine software measures (charging control, throttling, disabling background apps) with hardware measures (passive heatsinks, improved airflow, occasional breaks) for best results.

• Back up your data and test settings incrementally; small adjustments help find a stable balance between uptime and longevity.

This post covers a much more technical and involving way to run local LLMs on android via a terminal setup.

If you want to try an easy-to-use , less technical way, I have it covered in this latest blog post https://nkaushik.in/writing/top-4-ways-to-run-llm-locally-on-android-and-ios/

In the above post I go through 4 ways to run on LLMs on both Android and iOS devices.

When LLM models were first launched, we had to rely on the cloud versions, like ChatGPT or Gemini. However, things are changing for the better. We are now seeing a wave of new AI models being released every week, and most of them can run locally with good performance. This allows us to perform AI inference on edge or even mobile devices.

I have been running these models on my Windows machine using Ollama for a while, and even on my latest high-end Android phone with apps like Google AI Edge Gallery, PocketPal, and AnythingLLM. Everything has been running smoothly. I've used these models for tasks like describing images or helping me improve my writing. The speed of inference is quite fast, thanks to the latest and fastest mobile SoCs they are running on.

BUT…

I wanted to try something different. I had an old Android phone (OnePlus 3T) lying around, and I always wondered if I could find a use for it. So, this weekend, I did a quick proof of concept.

I tested whether I could run these new AI models locally on this old Android device, which doesn't have the most powerful hardware. Fortunately, tools like Unsloth, GGUF, and Llama.cpp are available.

In cases where we have limited RAM, the GGUF format of models lets us run them with much lower VRAM requirements while maintaining nearly the same accuracy.

To get started, I first needed to figure out how to run these models. Llama.cpp offers a CLI or server method to interact with your model. However, they provide pre-built binaries for Windows and Linux, not for Android, so we need to compile them for Android.

🛠️Installing Termux

To compile llama.cpp for Android, I needed a shell, and Termux was the solution. The latest version on the Play Store wasn't compatible with my device, but I managed to download and install the APK from F-Droid. With Termux running, I was ready to proceed to the next step of compiling llama.cpp..

👷🏼Building llama.cpp

This was quite straightforward, all I had to do was follow the steps:

https://github.com/ggml-org/llama.cpp/blob/master/docs/android.md#build-cli-on-android-using-termux

🏃🏻Running the model

Once llama.cpp was built, the binaries were placed in the bin folder under the build directory. Then, I could start the server using

./build/bin/llama-server -m model-path -c 2048 -n 4096 -—host 0.0.0.0 --port 8080

Since I was going to connect to this server from a different device, I had to set the host to 0.0.0.0.

I tried using 4 different models with Q4_K_M quantization:

• https://huggingface.co/unsloth/LFM2.5-1.2B-Instruct-GGUF

• https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking

• https://huggingface.co/unsloth/Qwen3-0.6B-GGUF

• https://huggingface.co/unsloth/gemma-3-1b-it-GGUF

I also had to adjust the -c and -n option values based on the available RAM.

Once the server was running, I needed a client to connect to it.

First, I configured the AnythingLLM app. Llama.cpp starts a server that is compatible with the Generic OpenAI spec. However, I often encountered issues with the app when trying to use the model, and it simply wouldn't work.

Then, I switched to Open-WebUI as a client, and it was an immediate success. As soon as I entered the details, it detected the model, and starting a chat with the model worked smoothly.

I tried running each model one by one to understand their performance. As expected, Qwen 3 0.6B was the fastest. The performance of these models was just okay. In some cases, I could get 10-18 tokens per second, while the Thinking models and larger models only produced 4-5 tokens per second. So, while they worked, the results weren't very impressive.

A key point to remember is that the OnePlus 3T was launched 10 years ago, so I wasn't expecting groundbreaking performance. For comparison, when I run local models on the Snapdragon 8 Elite chipset, I get over 30 tokens per second on the LFM2.5 Thinking model with Q8 quantization, along with almost instant Time to First Token(TTFT). This performance is better and more consistent with other models I ran on the same chipset.

Another downside of running these models on a mobile device is the heat. Extended use will always cause your device to start heating up.

In conclusion, running LLM models on older Android devices is a feasible endeavor, albeit with some limitations. By leveraging tools like Termux and llama.cpp, it's possible to compile and execute AI models locally, even on hardware that is not cutting-edge. While performance may not match that of newer devices, and issues such as heat generation and slower token processing rates are present, this approach offers a valuable way to repurpose older technology. It demonstrates the potential for AI applications to be more accessible and versatile, allowing users to explore AI capabilities without relying solely on high-end devices or cloud-based solutions.

I have over ten years of experience in the software industry, where I have worked with various organizations in both B2C and B2B sectors.

Over the years, I have worked with many technologies and tools that fascinated me. I began my journey with PHP and have since built Native Android Apps and React Native Apps. I've also had the chance to work with Ruby on Rails, Golang, and Python. Additionally, I have hands-on experience in maintaining services and infrastructure on both Google Cloud (GCP) and AWS, as well as with Kubernetes (k8s).

I currently work in the JavaScript/TypeScript ecosystem, specializing in React.JS & Node.js.

I possess extensive expertise in the frontend domain, which I recently leveraged to build a comprehensive Performance Testing platform from the ground up. This project encompassed not only the frontend but also the backend and the data engineering aspects. I undertook this ambitious endeavor with a very small team, yet we successfully implemented the entire platform. It is now widely used throughout the organization and serves as a crucial guardrail for performance-related checks across the entire company. The platform's integration has significantly enhanced our ability to monitor and optimize performance, ensuring that all our systems operate efficiently and effectively. This achievement has not only streamlined our processes but also reinforced the importance of performance testing within our organizational framework.

I enjoy building things and trying out new ideas. I've been working to keep up with the rapid advancements in the Generative AI space.

With time the things, a browser could do have also increased greatly (except IE11, switch to Firefox already). A webpage is no longer about some text and images, you can add a lot more power-hungry features to your webpage. You can create Games, Build Services like Google Docs, Browser Stack, and a lot more all running on a browser.

This post is going to talk about one of the browser’s features I had a chance to work with and the tech I explored for integrating it.

Problem Statement

The problem statement I had in front of me was to implement a screen-share feature where-in when a User is sharing the screen it would be possible for others to view it over the internet as a real-time video.

The first step was to break down this problem into smaller problems and build a prototype around it.
As I saw it, the problem had two pieces, first figuring out how to capture the screen, second, how to send it across to other users as a live video feed.
I’ll start with the Screen-share bit first.

Screen-share/Record

I knew websites already do it (you would’ve seen it in action on Zoom or Google Meet) but wasn’t aware How easy it is to do it on our own.

const promise = navigator.mediaDevices.getDisplayMedia(constraints);

The above line is all it takes to get started with screen-sharing. It is supported by all the latest browsers and you can read more on its support at https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getDisplayMedia

When you run the above code snippet you will get a Promise<MediaStream>, now it is up-to us to decide how we want to use this.

We can either share it with other people using WebRTC (more on it later) or even record it and download the video.

Example snippet to download the recorded video (taken from https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_Recording_API)

promise.then(stream => {
    var recordedChunks = [];

    console.log(stream);

    var options = { mimeType: "video/webm; codecs=vp8" };
    mediaRecorder = new MediaRecorder(stream, options);

    mediaRecorder.ondataavailable = handleDataAvailable;
    mediaRecorder.start();

    function handleDataAvailable(event) {
        console.log("data-available");
        if (event.data.size > 0) {
            recordedChunks.push(event.data);
            console.log(recordedChunks);
            download();
        } else {
        // ...
        }
    }

    function download() {
        var blob = new Blob(recordedChunks, {
            type: "video/webm"
        });
        var url = URL.createObjectURL(blob);

        var a = document.createElement("a");

        document.body.appendChild(a);
        a.style = "display: none";
        a.href = url;
        a.download = "test.webm";
        a.click();
        window.URL.revokeObjectURL(url);
    }

    // demo: to download after 9sec
    setTimeout(event => {
        console.log("stopping");
            mediaRecorder.stop();
        }, 9000);
    });
}

One thing to note about the above getDisplayMedia API is it behaves differently on browsers. For example, Chrome lets you share between a single tab/window/desktop, Firefox shares the whole window and Safari shares the entire desktop.

A limitation this API has is, it won’t let you share only a part of your webpage.

Now that we have solved half of the problem, let’s get to the second part.

WebRTC

If you haven’t heard about it, don’t worry, I will add resources for you to read. It is an interesting topic so you should give it a try.

WebRTC (Web Real-Time Communication) is a technology that enables Web applications and sites to capture and optionally stream audio and/or video media, as well as to exchange arbitrary data between browsers without requiring an intermediary.

The above definition is taken from MDN docs which captures the gist of what WebRTC is (https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API).

WebRTC is a very detailed topic and it won’t be easy to cover all that it provides in this article. I’d highly recommend going through this video on YouTube.

https://www.youtube.com/watch?v=p2HzZkd2A40

Based on the above video, we derive two important steps, we need to acquire a video/audio stream and then use WebRTC to communicate it with peers.

This video stream that we share can either be a stream from your webcam or any MediaStream from other sources.

If you recall, the screen-share stream we acquired is also a MediaStream which can be used with WebRTC.

So once we have set-up our WebRTC connection we can simply publish this Screen-share stream on it and make it available to peers in real-time.

Implementing WebRTC from scratch using the basic API can be a tedious task, there are various libraries and frameworks available for use. These libraries hides away the software complexity, but to ensure WebRTC works for 100% of the users you would also need to setup a server of your own.

PeerJS (https://peerjs.com) is one such example, which makes it easier for you to get started.

Various companies provide you with the infrastructure and SDK needed to get started with real-time video, for example, Twilio. This takes away both the software and hardware complexity and makes it easier for you to implement and support a wide variety of users and devices.

In my case, we were already utilizing Twilio for the video call feature and it also let us publish more MediaStream if needed for a user.

This made sharing screen-share with other users a breeze, as all it required was to acquire the screen-share stream and then publish it on the connection for a user.

Twilio also has a sample app that you can refer to for understanding how to do the above.

https://github.com/twilio/twilio-video-app-react

Conclusion

This was just one example of what can be achieved using these 2 pieces of technologies: WebRTC & getDisplayMedia.

The possibilities are limitless.

For example, You can make a piece of software which records your webcam as well as your screen to make a coding tutorial with few lines of code right from your browser.

A small experiment I did was to find a way to only share a part of webpage. For this I wrapped the area which I wanted to share with a distinctive color border and on other users before showing the video stream I simply used image processing to crop the pieces outside of the border. This isn’t a very great solution and is extremely memory intensive to do it in JavaScript so I won’t recommend it.

Special shout-out to Punit Gupta for helping me with this article.

Thanks for reading.
Do share your thoughts and suggestions.

I already have had some hands-on experience with Python, Java, PHP so this new thing had to be neither of these. Rust & Go are the 2 new buzz-words I kept hearing when it comes to Programming languages. I decided to explore these 2 languages and I was fascinated by Golang and after watching various videos of Why and How Golang was built I decided to start my journey with it.

As with any new language, I started with the “hello world” program. But to fully enjoy the language I needed to make something bit bigger.

I started exploring for ideas and then one day came up with a thought of how does GitHub works. We know it relies on git but what does it take to make something similar of my own and in-process get my hands dirty with Go.

This is the story of various topics I explored, blog posts & docs I read to learn something and share it.

Disclaimer: I am in no way expert on the below technologies, this post highlights the various things I learned while building the following project.

Goal

The aim of this project was to setup a GitHub like website where I could upload my code, using HTTP for git operations on remote repository. (This post won’t cover what it takes to support SSH based git remote operations, and would implement the most basic functionality.)

I wanted to explore Golang language and I would be using the code snippets or examples written in Golang but the core logic/idea would remain the same irrespective of any language.

The Basics

You can follow any guide to setup a basic HTTP server to handle various routes in your preferred web framework.

For my project I used gin (http://github.com/gin-gonic/gin) to setup some basic routes.

To make things simpler I wrote my routes in such a way that all the git operations on repository happens on a URL with this format. [http://domain.com/git/repo-name](http://domain.com/git/repo-name)
This means the remote URL for the local repository would look like above and you would use this URL to clone a repository and for other git operations as well.

Adding Functionality

Creating a Repository

POST /repo
Content-Type: application/json

{
    "name": "repo-name"
}

We know that to start a git repository we do git init . Running this command creates a .git folder with all the required files.

What happens when you need to create a repository on your server?

We utilize the same command but we pass an additional option --bare .
Passing this option creates a git repository without a working tree, this created repository is different as it will prevent any changes to occur on the remote repository directly and one can’t run the usual git commit command directly in this directory.

Try running the git init --bare command on your system and see the file structure, you will notice that it is similar to what .git directory looks like but instead of all these files/folder being present in .git folder it is present in the root of your folder where you ran the command.

git init --bare file structure

You can look more into the --bare option to understand it even better.

Ref: https://www.atlassian.com/git/tutorials/setting-up-a-repository/git-init

Now that we know how to setup a repository on server we can create an endpoint which will let us create new repository.

Code for such an endpoint could look like below. Here I have stripped out the various check I implemented before actually creating a repository.

Create Repository HTTP Route

I created a small utils package which I used to handle all git specific operations. Code for the CreateNewRepo function used in above snippet is below.

We now have a endpoint set-up which when hit will create a new repository for us.

Now let’s implement the main git operations.

Implementing Git Operations support

When we do git clone git push git pull on our terminal if the remote repository URL is HTTP based git internally uses HTTP as the mode to perform the required operations for these commands to work.

This is a key piece of information as all we need to do now is make sure our server support all the endpoints git will use and also handle the response and request for these endpoints.

Luckily there are already tools/libraries which does this and we won’t need to implement them from scratch.

For my project in Go I came across this project https://github.com/asim/git-http-backend/blob/master/server/server.go

The author of this project had already implemented all the necessary endpoints and the core functionality for each of them using go’s default HTTP server as the base.

Since I was using Gin I had to tweak the code to make it work specific to Gin framework, which was comparatively easy as all the Handler function needed was Request & Response .

That’s it.

We now have the required endpoints created and we can now take our code for a spin.

Let’s Execute

Let’s say if your server was running on port 8000 and you had created a repo named test .
In your local terminal you could do git clone http://localhost:8000/git/test and it would clone the repository (it would be blank if you hadn’t pushed anything).
You can now do git commit and push the changes, the changes would now be present on your remote repository.

References:

To read more in-depth about git I’d recommend the official docs (for in-depth)/Atlassian’s version (for brief intro)

Some more in-depth resources below:

Git - The Protocols

Git - pack-protocol Documentation

Next Steps

Git Hooks based events

I modified my server to setup a WebSocket connection on a route which looks like http://localhost:8000/ws/repo-name/ . This would now create a per repo based WebSocket connection between the clients and our server.
I could then use this connection to push various information to clients.

An example of such use-case is below:

Together with Git hooks (post-receive) I can now push messages on the WebSocket Connection to the clients whenever a new commit/ref is updated on the remote repository.
This would allow me to setup a front-end in future which can listen to these events and show a message to the user.

You can read more about the Git hooks here

Git - Git Hooks

For making the above feature you would need to setup Server Side Hooks.

Authentication

To implement basic authentication where a different user can not push to another user’s repository you can use the pre-receive git hook on server side and add the authentication logic and block the operations if access is denied. (There may be other ways to implement this feature this is something I thought of and haven’t implemented yet.)

This is my first-time writing a blog post, feedback is appreciated.

Thanks.