The biggest symptoms were:

• A Google performance score around 60 on the landing page

• A very large production bundle

• Heavy CSS and font payloads

• Too much non-critical work happening during initial load

This was the warning I getting from vite while running build command.

vite v6.4.1 building for production...
✓ 2527 modules transformed.
dist/index.html
(!) Some chunks are larger than 500 kB after minification. Consider:
- Using dynamic import() to code-split the application
- Use build.rollupOptions.output.manualChunks to improve chunking: https://rollupjs.org/configuration-options/#output-manualchunks
- Adjust chunk size limit for this warning via build.chunkSizeWarningLimit.
✓ built in 9.64s

This write-up focuses only on the frontend side. API performance was intentionally left out.

The Problem

From the original production build:

• Main JS bundle: 1,841.83 kB minified

• Main CSS bundle: 269.42 kB

• Gzipped JS: 590.54 kB

• Gzipped CSS: 61.27 kB

That meant the browser had to parse, download, and evaluate far more than the landing page actually needed.

Strategy

I kept the plan simple:

1. Reduce the critical path

2. Load only what the current page needs

3. Push non-essential work after first paint

4. Keep regressions low by changing loading strategy first, not rewriting features

What I Changed

1. Route-level code splitting

The landing page should not pay the cost of planner screens, task editing flows, integrations, and dashboard code.

I changed the app so major routes load on demand instead of being pulled into the first page load.

const routes = [
  { path: "/", component: () => import("...HomePage...") },
  { path: "/app", component: () => import("...WorkspaceView...") },
  { path: "/settings", component: () => import("...SettingsView...") },
];

2. Lazy-loading expensive UI inside authenticated screens

Even inside the app, some heavy UI only appears after user interaction.

Examples:

• task edit modal

• integration panels

• filter sidebar

• device-specific overlays

So instead of bundling them into the initial workspace render, I loaded them only when needed.

const HeavyModal = defineAsyncComponent(() => import("...TaskEditor..."));

you can read more about defineAsyncComponent on official docs.

3. Deferring analytics and monitoring

Analytics, session tracking, and error monitoring are useful, but they do not need to compete with first paint.

I moved them behind a delayed, idle-time startup.

runAfterLoadAndIdle(() => {
  setTimeout(() => {
    initAnalytics();
    initMonitoring();
    loadThirdPartyScript();
  }, 4000);
});

This was one of the highest-leverage changes for Lighthouse.

4. Font loading discipline

The previous build was shipping a wide set of font families and language subsets up front.

I changed that strategy to:

• load only base fonts first

• load landing-page accent fonts only on the landing page

• load theme-specific fonts only when that theme is active

• use narrower subsets instead of pulling everything by default

loadBaseFonts();
loadLandingFonts();
loadThemeFonts(currentTheme);

5. Reducing work on the landing page itself

The landing page also had runtime work that looked nice but did not need to happen immediately.

I optimized that by:

• delaying visual effects until idle time

• keeping below-the-fold sections out of the initial rendering cost

• adding image dimensions to reduce layout work

• preserving design while trimming early execution

Benchmarks

Build Footprint

Lighthouse Snapshot

Notes:

• The new JS number reflects the critical landing path, not the entire app footprint.

• Large authenticated features still exist, but they are no longer loaded up front on the landing page.

Why This Worked

The biggest lesson was straightforward:

I did not need less product. I needed less product in the first request.

Most performance issues were not caused by one huge mistake. They came from many reasonable frontend decisions stacking together:

• eager route imports

• global loading of rare UI

• early analytics boot

• broad font loading

• decorative runtime work too early

Once I treated the landing page like a focused entry experience instead of the whole app, performance improved quickly.

Takeaways

• Start with the critical path, not micro-optimizations

• Route splitting gives outsized wins when a landing page and app shell live together

• Third-party scripts should earn their place in the first few seconds

• Font strategy matters more than many teams think

• Good performance work is often about loading order, not feature removal

Final Summary

I improved page speed by shrinking the initial frontend payload, deferring non-essential work, and keeping heavy product features out of the landing-page critical path.

The result was a much lighter first load, better rendering behavior, and a local Lighthouse score that moved from the low-performance range into the 90+ range without redesigning the product.

Before you go:

if you're someone struggling with managing time, then checkout this light weight but powerful web app Lazy Planner
i made this for myself first because i couldn't find the one that better suits my need but i believe if it's useful for me, it could be useful for more people too.

anyway i was exploring and i found this package pyreverse. it’s so easy to generate a mind map of imports.

use this cmd in root folder but make sure your project’s root folder have __init__.py file.
then run

pyreverse -o png .

this will need graphviz library to generate png images which u can install using apt or brew install graphviz but u can also use other format to export. check pyreverse docs for more details about different formats.

understanding this graph is simple too. the arrows should go upward and there shouldn’t be arrows pointing at each other. if they are pointing at each other, you probably already have circular import issues and errors will be shown when you run the code. but if error isn’t raised, still it’s a good idea to rethink about project imports as you’re close to circular import errors.

Why Password Hashing Matters

Storing passwords in plain text is an absolute no-go. If your database gets compromised, plain passwords put your users at immense risk. Instead, Django uses password hashing, which irreversibly transforms a password into a fixed-length string. This way, even if someone accesses your database, they won’t see the actual passwords.

What Algorithm Does Django Use?

By default, Django uses the PBKDF2 algorithm with SHA-256 as its hashing method. PBKDF2 (Password-Based Key Derivation Function 2) applies the SHA-256 hash function many thousands of times — typically 260,000 iterations or more — making brute-force attacks computationally expensive and slow.

Django also supports alternative hashing algorithms like Argon2, BCryptSHA256, and Scrypt, which you can configure in your settings if desired.

The Role of Salt in Password Hashing

A core ingredient in Django’s password hashing is the salt: a unique, randomly generated string added to each password before hashing. The purpose of this salt is to guarantee that even if two users have the same password, their hashed values will differ. This protects against rainbow table attacks, which rely on precomputed hashes to reverse-engineer passwords.

Importantly, Django does not keep the salt secret. Instead, it stores the salt alongside the hashed password in the database. This allows Django to use the same salt again during login verification, ensuring accurate comparison.

What Does a Hashed Password Look Like?

When you inspect Django’s user database, passwords are stored as strings with a specific format:

algorithm$iterations$salt$hashed_password

For example:

textpbkdf2_sha256$260000$ks9a7bsZqP11$hbN5XwLoKA54tJdIQcVepXI/vXftN4Br2nqSiwDrTyM=

Breaking it down:

• pbkdf2_sha256: The hashing algorithm used

• 260000: Number of hash iterations

• ks9a7bsZqP11: The random salt

• The last part is the actual hashed password

How Password Verification Works During Login

When a user attempts to log in, Django does the following:

1. It fetches the stored hashed password string from the database.

2. Extracts the components: algorithm, iteration count, and salt.

3. Re-hashes the password entered by the user with those exact parameters.

4. Compares the freshly hashed value with the stored hash.

5. If they match, the password is correct. If not, access is denied.

Because the salt is stored with the hash, Django can reproduce the hashing process precisely.

What About the SECRET_KEY?

It’s a common misconception that Django’s SECRET_KEY is involved in password hashing. It isn’t. The secret key is primarily used for cryptographic signing elsewhere in Django (e.g., sessions, CSRF tokens) and is not part of password hashing.

Customizing Password Hashing

Django’s flexibility shines here. You can customize the PASSWORD_HASHERS setting to choose your preferred algorithms. For example, you can switch to Argon2 for stronger security, or add multiple hashers to support legacy passwords during a transition. New passwords will be hashed with the first hasher in your list.

Summary

• Django uses PBKDF2 with SHA-256 and a high iteration count by default.

• Each password is uniquely salted with a random string stored alongside the hash.

• Passwords are stored in a format combining algorithm, iteration count, salt, and hashed password.

• Login verification reuses the stored salt and parameters for hashing the entered password.

• Django’s secret key is unrelated to password hashing.

• You can customize which hashing algorithms Django uses via settings.

Understanding these mechanics ensures you appreciate Django’s secure design and can confidently manage user authentication in your projects.

This guide provides a secure, production-ready deployment process for the Focus Timer application on a Linux VPS. It follows best practices at each step and includes brief explanations to help beginners understand.

Prerequisites

• A fresh Ubuntu/Debian VPS (18.04+ or 20.04+).

• Root or sudo access.

• A domain name pointing to your VPS.

• Basic familiarity with Linux shell.

1. VPS Setup

1.1 Create Unprivileged User

Why? Running services as non-root improves security.

# Add a user, no password login, add to sudo group
adduser --disabled-password --gecos "" focususer
usermod -aG sudo focususer
# Set up SSH access for the new user
# Copy root's authorized keys to the new user
mkdir -p /home/focususer/.ssh
cp /root/.ssh/authorized_keys /home/focususer/.ssh/authorized_keys
# Set ownership and permissions
chown -R focususer:focususer /home/focususer/.ssh
chmod 700 /home/focususer/.ssh
chmod 600 /home/focususer/.ssh/authorized_keys

Explanation:

• adduser: Creates a new Linux user with default settings.

• --disabled-password: Prevents setting a login password (forces SSH key auth only).

• --gecos "": Supplies empty fields for the user's full name and contact info.

• usermod -aG sudo: Appends (-a) the user to the sudo group, allowing administrative commands.

1.2 Secure SSH Access

Why? Prevent unauthorized root or password-based logins.

# On your local machine, generate SSH key if needed:
ssh-keygen -t ed25519 -C "your_email@example.com"

# Copy public key to server:
ssh-copy-id focususer@your.domain.com

On server, edit /etc/ssh/sshd_config:

  Port 22
  PermitRootLogin no
  PasswordAuthentication no
  ChallengeResponseAuthentication no
  UsePAM yes
  sudo systemctl reload ssh

Explanation:

• ssh-keygen -t ed25519: Generates a modern, high-security Ed25519 key pair.

• -C "comment": Adds an identifying comment (often your email).

• ssh-copy-id: Installs your public key on the remote server's ~/.ssh/authorized_keys file, allowing key-based logins.

• PermitRootLogin no: Disables SSH login as root, forcing administrative access via a less-privileged user plus sudo.

• PasswordAuthentication no: Turns off password-based logins entirely; only key-based auth is allowed.

• ChallengeResponseAuthentication no: Disables keyboard-interactive (challenge-response) methods, such as one-time passwords or other PAM-driven prompts, preventing any fallback from key-based authentication.

  • Without this setting, SSH could invoke PAM's challenge modules (e.g. OTP or custom scripts) that might allow weaker or interactive authentication paths.

• UsePAM yes: Enables Pluggable Authentication Modules (PAM) integration.

  • Even with password logins disabled, PAM handles account and session management after a successful key login.

  • PAM modules enforce additional security policies (e.g. account expiration, lockouts, resource limits, logging).

  • Ensures that system-wide policies (fail2ban, pam_tally2, custom modules) can apply to every SSH session.

• systemctl reload ssh: Applies the above SSH daemon changes without dropping existing connections.

1.2.1. Logging in as focususer

Why? Operating as a non-root user mitigates the risk of accidental system-wide changes.

# Exit the current root SSH session:
exit

# From your local machine, SSH back in as the unprivileged user:
ssh focususer@your.domain.com

Explanation:

• exit: Ends the current SSH session as root and returns you to your local shell.

• ssh focususer@your.domain.com: Initiates a new SSH session using your key for the focususer account.

1.3 Firewall & Fail2ban

Why? Limit open ports and block brute-force attempts.

# Install UFW & Fail2ban
sudo apt update && sudo apt install -y ufw fail2ban

# Basic UFW rules
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable

# Configure Fail2ban (e.g., /etc/fail2ban/jail.local)
sudo tee /etc/fail2ban/jail.local <<EOF
[DEFAULT]
bantime  = 3600
findtime = 600
maxretry = 5

[sshd]
enabled = true
port    = 22
filter  = sshd
logpath = /var/log/auth.log
EOF
sudo systemctl restart fail2ban

Explanation:

• ufw default deny incoming: Blocks all incoming traffic by default.

• ufw default allow outgoing: Allows all outbound traffic.

• ufw allow OpenSSH: Opens SSH port (usually 22).

• ufw allow 'Nginx Full': Opens HTTP (80) and HTTPS (443) for web traffic.

• ufw enable: Activates the firewall with the defined rules.

• fail2ban: Monitors log files and bans IPs after too many failed login attempts.

• In jail.local:

  • bantime: Duration (1h) to ban offenders.

  • findtime: Time window (10m) to track failures.

  • maxretry: Max allowed failures (5) before ban.

1.4 Automatic Security Updates

Why? Keep critical packages patched.

sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure unattended-upgrades

Explanation:

• unattended-upgrades: Automatically downloads and installs security updates.

• dpkg-reconfigure unattended-upgrades: Opens a configuration prompt to enable auto-updates.

2. System Dependencies

Install tools required for building and deployment.

sudo apt install -y git build-essential curl libpq-dev

Explanation:

• git: Version control system to clone your repository.

• build-essential: Installs GCC, make, and other build tools for compiling native extensions.

• curl: Tool to transfer data from or to a server (used for downloading scripts).

• libpq-dev: Development headers for PostgreSQL, required by psycopg2 Python package.

3. Database: PostgreSQL

sudo apt install -y postgresql postgresql-contrib

# Create database and user
check ./setup.md for PostgreSQL setup

# Secure PostgreSQL: only listen locally and enforce password auth
sudo sed -i "s/#listen_addresses = 'localhost'/listen_addresses = 'localhost'/" /etc/postgresql/*/main/postgresql.conf
# Only allow local socket and password auth for TCP
host    all             all             127.0.0.1/32            md5
sudo systemctl restart postgresql

Brief: PostgreSQL is reliable and scalable for production.

4. Redis for Celery

sudo apt install -y redis-server
sudo systemctl enable --now redis-server.service

# Secure Redis: bind only to localhost, enable protected mode, optional password
sudo sed -i "s/^bind .*/bind 127.0.0.1 ::1/" /etc/redis/redis.conf
sudo sed -i "s/^protected-mode no/protected-mode yes/" /etc/redis/redis.conf
sudo sed -i "s/# requirepass foobared/requirepass StrongRedisPassw0rd/" /etc/redis/redis.conf
sudo systemctl restart redis

Brief: Redis acts as broker and result backend for Celery.

5. Codebase Deployment

5.1 Clone Repository

# Switch to DIR where you want to keep your project
# ( using a sample project name, replace with your actual repo )
git clone https://github.com/your-repo/focus-timer-django-vue.git
cd focus-timer-django-vue

5.2 Environment Variables

Why? Keep secrets out of source code.

# In project root, create .env
touch .env
# Populate .env with necessary variables
# Secure .env file permissions
chmod 600 .env

6. Python Virtual Environment

cd backend
python3 -m venv .venv
source .venv/bin/activate
pip install uv
pip install -r requirements.txt

7. Frontend Build (Vue.js)

cd ../frontend-vue
# install node
curl -fsSL https://raw.githubusercontent.com/mklement0/n-install/stable/bin/n-install | bash -s 22

npm run build

Brief: Outputs static assets in dist/.

8. Django Migrations & Static Files

cd ../backend
source .venv/bin/activate
python manage.py migrate
python manage.py collectstatic --noinput

9. Application Server: Gunicorn

pip install gunicorn

Create /etc/systemd/system/gunicorn.service:

[Unit]
Description=Gunicorn daemon for Focus Timer
After=network.target

[Service]
User=focususer
Group=www-data
WorkingDirectory=/home/focususer/focus-timer-django-vue/backend
ExecStart=/home/focususer/focus-timer-django-vue/.venv/bin/gunicorn \
  backend.wsgi:application \
  --bind 127.0.0.1:8000 \
  --workers 4
EnvironmentFile=/home/focususer/focus-timer-django-vue/.env
Restart=on-failure
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now gunicorn

10. Background Workers: Celery & Beat

• Create /etc/systemd/system/celery.service:

[Unit]
Description=Celery worker service for Focus Timer
After=network.target

[Service]
Type=simple
User=focususer
Group=www-data
WorkingDirectory=/home/focususer/focus-timer-django-vue/backend
EnvironmentFile=/home/focususer/focus-timer-django-vue/.env
ExecStart=/home/focususer/focus-timer-django-vue/.venv/bin/celery -A backend worker \
  --loglevel=info
Restart=on-failure

[Install]
WantedBy=multi-user.target

• Create /etc/systemd/system/celery-beat.service:

[Unit]
Description=Celery Beat scheduler for Focus Timer
After=network.target

[Service]
Type=simple
User=focususer
Group=www-data
WorkingDirectory=/home/focususer/focus-timer-django-vue/backend
EnvironmentFile=/home/focususer/focus-timer-django-vue/.env
ExecStart=/home/focususer/focus-timer-django-vue/.venv/bin/celery -A backend beat \
  --loglevel=info  \
  --scheduler django_celery_beat.schedulers:DatabaseScheduler
Restart=on-failure

[Install]
WantedBy=multi-user.target

• Enable & start both services:

sudo systemctl daemon-reload
sudo systemctl enable --now celery celery-beat

Explanation:

• Type=forking/simple: ensures proper startup and process tracking.

• --detach: runs worker/beat in background.

• Restart=on-failure: automatically recovers from crashes.

11. Reverse Proxy: Nginx

Install and configure /etc/nginx/sites-available/focus-timer:

server {
    listen 80;
    server_name tymr.online www.tymr.online;

    root /home/focususer/focus-timer-django-vue/frontend-vue/dist;
    index index.html;

    # incase using websockets
    location /ws/ {
        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $remote_addr;

        proxy_read_timeout 86400;
    }

    location / {
        try_files $uri $uri/ /index.html;
    }
    location /static/ {
        alias /home/focususer/focus-timer-django-vue/backend/static/;
    }
    location /api/    { proxy_pass http://127.0.0.1:8000/api/; include proxy_params; }
    location /auth/   { proxy_pass http://127.0.0.1:8000/auth/; include proxy_params; }
    location /admin/  { proxy_pass http://127.0.0.1:8000/admin/; include proxy_params; }

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    client_max_body_size 10M;
}

Enable and reload:

sudo ln -s /etc/nginx/sites-available/focus-timer /etc/nginx/sites-enabled/
sudo nginx -t
sudo ln -s /etc/nginx/sites-available/focus-timer-redirect /etc/nginx/sites-enabled/
sudo systemctl reload nginx

Explanation:

• Security headers prevent clickjacking, MIME-sniffing, enforce HTTPS.

• client_max_body_size: limits upload size to mitigate DoS.

• HTTP->HTTPS redirect ensures all traffic is encrypted.

12. SSL/TLS with Let's Encrypt

sudo apt install -y certbot python3-certbot-nginx
sudo certbot --nginx -d tymr.online -d www.tymr.online

Brief: Provides free, auto-renewing certificates.

13. Monitoring & Logging

• Use journalctl -u gunicorn -f and journalctl -u celery -f.

• Set up Logrotate for Gunicorn logs if needed.

• Consider external monitoring (Prometheus/Grafana, UptimeRobot).

14. Automated Backups

• Write a cron job for nightly PostgreSQL dumps:

    sudo crontab -u focususer -e
    # Add:
    0 2 * * * pg_dump -U focusdbuser focusdb | gzip > ~/backups/db-$(date +\%F).sql.gz
  

• Secure backup storage (offsite or S3).

15. Routine Maintenance

• Keep OS & packages updated:

    sudo apt update && sudo apt upgrade -y
  

• Review logs, monitor disk and memory.

16. Nginx permissions

Why? Nginx needs access to static files and directories.

# Ensure Nginx can read static files
sudo chmod o+x /home/focususer
sudo chmod -R o+rX /home/focususer/focus-timer-django-vue/frontend-vue/dist
sudo chmod -R o+rX /home/focususer/focus-timer-django-vue/backend/static
sudo chmod -R o+rX /home/focususer/focus-timer-django-vue/backend/media
sudo chmod -R o+x /home/focususer/focus-timer-django-vue/frontend-vue
chmod +x /home/focususer/focus-timer-django-vue/.venv/bin/gunicorn
chmod +x /home/focususer/focus-timer-django-vue/.venv/bin/celery
chmod u+x restart_all.sh

Few helpful commands to restart services after deployment

1. Reload all systemd unit files (after any edits)

sudo systemctl daemon-reload

2. Restart your Django app (Gunicorn)

sudo systemctl restart gunicorn sudo systemctl status gunicorn # check exit status immediately journalctl -u gunicorn -f # live logs

3. Restart Celery worker & beat

sudo systemctl restart celery celery-beat sudo systemctl status celery # check worker status sudo systemctl status celery-beat journalctl -u celery -f # live worker logs journalctl -u celery-beat -f # live beat logs

4. Restart Nginx (reverse proxy & static files)

sudo systemctl restart nginx sudo systemctl status nginx sudo journalctl -u nginx -f

5. (Optional) Restart backing services

sudo systemctl restart redis-server postgresql sudo systemctl status redis-server postgresql sudo journalctl -u redis-server -f sudo journalctl -u postgresql -f

6. (Optional) I made a script to restart all services at once. Feel free to tweak it as it fits your needs.

#!/usr/bin/env bash

# Script to restart all Focus Timer services

set -euo pipefail

# Parse options
SKIP_FRONTEND=false
for arg in "$@"; do
  case $arg in
    --skip-frontend)
      SKIP_FRONTEND=true
      ;;
  esac
done
echo "Installing backend requirements..."
.venv/bin/uv pip install -r backend/requirements.txt

echo "Applying database migrations..."
.venv/bin/python backend/manage.py migrate --noinput

echo "Collecting static files..."
.venv/bin/python backend/manage.py collectstatic --noinput
if [ "$SKIP_FRONTEND" = false ]; then
  echo "Building frontend..."
  ( cd frontend-vue && npm ci && npm run build )
else
  echo "Skipping frontend build due to --skip-frontend option"
fi

echo "Reloading systemd daemon..."
sudo systemctl daemon-reload

services=(
  gunicorn
  celery
  celery-beat
  nginx
  redis-server
  postgresql
)

for service in "${services[@]}"; do
  echo "Restarting $service..."
  sudo systemctl restart "$service"
done

echo "Waiting for services to settle..."
sleep 2

echo "Services status:"
for service in "${services[@]}"; do
  echo "===== $service ====="
  sudo systemctl status "$service" --no-pager
done

echo "All services restarted."

Save this script as deploy.sh and make it executable:

chmod +x deploy.sh

1. What Is Fail2Ban?

• Guard Dog: Monitors log files like a watchful puppy.
• Jails: Special zones around each door (service) where the guard listens.
• Filters: Magic spells (regular expressions) that spot naughty knocks (bad actions).
• Actions: When a robot misbehaves too much, the guard slams the door shut by adding a firewall rule.

2. Why Use Fail2Ban with Django?

1. Protect the Admin Tower
   The /admin/ page is a treasure room. Don’t let robots guess passwords!
2. Guard the Login Gate
   Your API login endpoint (/auth/login/) needs extra eyes.
3. Stop Crawler Floods
   Bots triggering hundreds of 404 pages can overwhelm the castle.
4. Layered Security
   Compliments your Django settings.py security (SSL redirect, HSTS, X-Frame-Options).

3. Setting Up Fail2Ban Step-by-Step

Step 3.1: Install Your Guard

sudo apt update
sudo apt install -y ufw fail2ban

• UFW: The castle’s drawbridge controller (firewall).
• Fail2Ban: The guard dog watching your logs.

Step 3.2: Configure the Drawbridge (UFW)

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow OpenSSH            # Let knights in via SSH
sudo ufw allow 'Nginx Full'       # Let visitors reach your website (HTTP/HTTPS)
sudo ufw enable

Step 3.3: Create the Jail Configuration

Create /etc/fail2ban/jail.local and add:

[DEFAULT]
bantime  = 3600        # Ban duration: 1 hour
findtime = 600         # Look back: 10 minutes
maxretry = 5           # After 5 bad tries, ban

[sshd]
enabled = true
port    = ssh
filter  = sshd
logpath = /var/log/auth.log

[django-login]
enabled  = true
port     = http,https
filter   = django-login
logpath  = /var/log/nginx/access.log

• bantime: How long the misbehaver is kicked out.
• findtime: Time window to count failures.
• maxretry: Number of strikes before the ban.

Step 3.4: Write Your Magic Spell (Filter)

Create /etc/fail2ban/filter.d/django-login.conf:

[Definition]
# Spot HTTP 401 responses on /auth/login/
failregex = ^<HOST> -.*POST /auth/login/.*" 401
ignoreregex =

This tells Fail2Ban: “Whenever you see a 401 Unauthorized on a login POST, count it!”

Step 3.5: Start the Guard Dog

sudo systemctl restart fail2ban

4. Testing Your Guard

1. Check All Jails

   sudo fail2ban-client status
   

2. Inspect the Django-Login Jail

   sudo fail2ban-client status django-login
   

3. Simulate Robot Knocks

   for i in {1..6}; do
     curl -s -o /dev/null -w "%{http_code}" \
       -X POST https://your-domain.com/auth/login/ \
       -H "Content-Type: application/json" \
       -d '{"username":"bad","password":"bad"}'
   done
   

   After 5 “bad” tries, your IP should appear in the banned list!

5. Extra Security Tips

• Protect Your Logs: Only root can edit /var/log/* so robots can’t erase evidence.
• Combine with Django Security: In settings.py, enable:
  • SECURE_SSL_REDIRECT = True
  • HSTS (SECURE_HSTS_SECONDS)
  • X_FRAME_OPTIONS = "DENY"
• Add More Jails:
  • nginx-http-auth (blocks repeated 401 basic-auth failures)
  • nginx-404 (blocks too many broken-page hits)
  • nginx-limit-req (stops traffic floods)
• Tweak Thresholds: Adjust bantime, findtime, maxretry based on real traffic.
• Email Alerts: Use action_mwl to notify you on each ban.

6. You Did It!

🎉 You now have a trusty Fail2Ban guard watching your Django castle. No more pesky robots breaking in! Keep learning, keep coding, and enjoy your safe server.

Content Security Policy (CSP) is a security standard that helps prevent cross-site scripting (XSS), clickjacking, and other code-injection attacks by whitelisting trusted sources of content. When correctly configured, CSP ensures that only approved scripts, styles, images, and connections can be loaded by the browser.

Why CSP Matters

Without a strict CSP, an attacker could inject malicious scripts into your application, steal user credentials, or redirect users to phishing pages:

flowchart LR
  subgraph Attacker
    A1[Craft malicious payload] --> A2[Host exploit page or social-engineer user]
  end

  subgraph Server
    B1[Receives request with payload]
    B2[Writes to DB or reflects it back]
    B1 --> B2
  end

  subgraph VictimBrowser
    C1[Fetch page from Server]
    C1 --> C2[Parses HTML including attacker’s `<script>`]
    C2 --> C3[Executes malicious JS]
    C3 --> C4[Steals cookies or redirects to phishing site]
  end

  A2 --> B1
  B2 --> C1

How CSP Works

When the browser receives HTML along with a CSP header, it enforces each directive on every resource request:

flowchart TD
  1[Start: Browser requests page] --> 2[Receive HTML + CSP header]
  2 --> 3[Parse HTML]
  3 --> 4{Resource Tag?}
  4 -->|Yes| 5[Identify Resource Type]
  5 --> 6[Lookup CSP directive for that type]
  6 --> 7{URL in whitelist?}
  7 -->|Yes| 8[Allow load]
  7 -->|No| 9[Block load & log warning]
  4 -->|No| 10[Continue rendering]
  8 --> 10
  9 --> 10
  10 --> 11[Page finishes rendering]
  11 --> 12[End]

Browser Enforcement Flow

This diagram shows how the browser enforces CSP on resources like scripts, styles, images, and XHR/fetch requests:

flowchart LR
  subgraph Server
    A[Serve index.html] -->|Include CSP header| B((HTTP response))
  end

  subgraph Browser
    B --> C{Read CSP header}
    C --> D[Enforce rules on scripts/styles/images/connect]
    D -->|Allowed| E[Load resource]
    D -->|Blocked| F[Drop resource + warn]
  end

  subgraph Page
    E --> G[App JS runs]
    F --> G
  end

Configuring CSP in Nginx for Vue.js

In your Nginx server block for the Vue.js static build, add a CSP header:

server {
    listen 80;
    server_name yourdomain.com;

    root /var/www/vue-dist;
    index index.html;

    add_header Content-Security-Policy "
      default-src 'self';
      script-src 'self' https://cdn.jsdelivr.net;
      style-src 'self' 'unsafe-inline';
      img-src 'self' data:;
      font-src 'self' https://fonts.gstatic.com;
      connect-src 'self' https://api.yourdomain.com;
    " always;

    location / {
        try_files $uri $uri/ /index.html;
    }
}

• default-src: Fallback for all resource types.
• script-src: Whitelist your own bundle and trusted CDNs.
• style-src: Allow inline styles for runtime libraries like Floating-UI.
• img-src: Permit data URIs for base64 images.
• font-src: Load web fonts from trusted providers.
• connect-src: Ensure the SPA can communicate with the Django API.

Configuring CSP in Django

Use the django-csp middleware or a custom header:

Using django-csp

# settings.py
MIDDLEWARE = [
    # ...
    'csp.middleware.CSPMiddleware',
]

CSP_DEFAULT_SRC = ("'self'",)
CSP_SCRIPT_SRC  = ("'self'", "https://cdn.jsdelivr.net")
CSP_STYLE_SRC   = ("'self'", "'unsafe-inline'")
CSP_IMG_SRC     = ("'self'", "data:")
CSP_CONNECT_SRC = ("'self'", "https://api.yourdomain.com")

Custom Middleware

# middleware.py
from django.utils.deprecation import MiddlewareMixin

class ContentSecurityPolicyMiddleware(MiddlewareMixin):
    def process_response(self, request, response):
        policy = (
            "default-src 'self'; "
            "script-src 'self' https://cdn.jsdelivr.net; "
            "style-src 'self' 'unsafe-inline'; "
            "img-src 'self' data:; "
            "connect-src 'self' https://api.yourdomain.com;"
        )
        response['Content-Security-Policy'] = policy
        return response

Note: If your Django project only serves API endpoints (no HTML responses or templates), you typically don't need to configure CSP on the Django side. Simply enforce CSP at your Nginx or frontend layer.

Inline Script Handling

When an attacker tries to inject inline <script> tags via URL parameters or form inputs, CSP will block execution:

flowchart LR
  A[Victim visits /page?msg=PAYLOAD] --> B[Browser requests HTML + CSP header]
  B --> C[Browser parses HTML]
  C --> D[Finds inline `<script>` from PAYLOAD]
  D --> E{script-src allows inline?}
  E -->|No| F[Block execution + throw console error]
  E -->|Yes| G[Would execute script]
  F --> H[User safe]

Testing & Troubleshooting

1. Check the browser console for “Refused to load” errors.
2. Whitelisting: Add missing domains in the right directive (e.g., connect-src vs. script-src).
3. Certificate permissions: Ensure Nginx can read SSL cert files if using HTTPS.
4. Disable Rocket Loader (Cloudflare) if you must avoid 'unsafe-inline' for scripts.

Conclusion

Implementing a strict Content Security Policy via Nginx and Django dramatically reduces the risk of XSS and resource injection attacks. By carefully whitelisting only trusted sources and monitoring violations, you harden both your Vue.js frontend and Django backend against a wide range of threats.

we will use sqlite online backup API for this. you can read more about it here https://www.sqlite.org/backup.html

The Online Backup API was created to address these concerns. The online backup API allows the contents of one database to be copied into another database file, replacing any original contents of the target database. The copy operation may be done incrementally, in which case the source database does not need to be locked for the duration of the copy, only for the brief periods of time when it is actually being read from. This allows other database users to continue without excessive delays while a backup of an online database is made.

System Components

1. Backup Creation

• Uses SQLite's Online Backup API

• Creates timestamped backups while the database is in use

• Maintains atomic consistency during backup

• Stores backups in a configured directory

apps/realtime_timer/management/commands/backup_database.py

import os
import sqlite3
from datetime import datetime
from django.core.management.base import BaseCommand
from django.conf import settings
import logging

logger = logging.getLogger(__name__)

class Command(BaseCommand):
    """
    Django management command to create SQLite database backups.
    Uses SQLite's built-in online backup API which allows creating backups
    while the database is in use.

    Usage:
        python manage.py backup_database

    The backup will be stored in the BACKUP_DIRECTORY specified in settings.SQLITE_BACKUP
    with a timestamp in the filename.
    """

    help = 'Creates a backup of the SQLite database using the online backup API'

    def handle(self, *args, **options):
        try:
            # Create backup directory if it doesn't exist
            # Uses the path specified in settings.SQLITE_BACKUP['BACKUP_DIRECTORY']
            backup_dir = os.path.join(settings.BASE_DIR, 'backups')
            os.makedirs(backup_dir, exist_ok=True)

            # Create a timestamp for unique backup filename
            # Format: YYYYMMDD_HHMMSS
            timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')

            # Get the path of the current database from Django settings
            db_path = settings.DATABASES['default']['NAME']

            # Generate the backup filepath with timestamp
            backup_path = os.path.join(backup_dir, f'backup_{timestamp}.db')

            # Open connection to the source (current) database
            source = sqlite3.connect(db_path)

            # Create a new connection for the backup database
            backup = sqlite3.connect(backup_path)

            # Use SQLite's backup API to create the backup
            # The backup() method is atomic and will maintain consistency
            # even if the database is being written to during backup
            with source, backup:
                source.backup(backup)
                logger.info(f"Database backup created successfully at {backup_path}")
                self.stdout.write(
                    self.style.SUCCESS(f'Successfully created backup at {backup_path}')
                )

        except Exception as e:
            logger.error(f"Database backup failed: {str(e)}")
            self.stdout.write(
                self.style.ERROR(f'Backup failed: {str(e)}')
            )

2. Backup Management

• Maintains a configurable number of recent backups

• Automatically removes older backups

• Includes backup rotation policies

• Prevents disk space issues

apps/realtime_timer/management/commands/manage_backups.py

import os
import glob
from datetime import datetime, timedelta
from django.core.management.base import BaseCommand
from django.conf import settings
import logging
from django.core.management import call_command

logger = logging.getLogger(__name__)

class Command(BaseCommand):
    """
    Django management command to manage SQLite database backups.
    This command:
    1. Removes old backups exceeding the maximum count
    2. Verifies the integrity of remaining backups

    Usage:
        python manage.py manage_backups

    Configuration is read from settings.SQLITE_BACKUP:
        BACKUP_DIRECTORY: Where backups are stored
        BACKUP_MAX_COUNT: Maximum number of backups to keep
    """

    help = 'Manages SQLite database backups by removing old ones'

    def handle(self, *args, **options):
        try:
            # Get backup settings from Django settings
            backup_dir = settings.SQLITE_BACKUP['BACKUP_DIRECTORY']
            max_backups = settings.SQLITE_BACKUP['BACKUP_MAX_COUNT']

            # Get all backup files and sort them by modification time
            # newest files first
            backup_files = glob.glob(os.path.join(backup_dir, 'backup_*.db'))
            backup_files.sort(key=os.path.getmtime, reverse=True)

            # If we have more backups than the maximum allowed
            if len(backup_files) > max_backups:
                # Remove the oldest backups (those beyond max_backups)
                for old_backup in backup_files[max_backups:]:
                    os.remove(old_backup)
                    logger.info(f"Removed old backup: {old_backup}")
                    self.stdout.write(
                        self.style.SUCCESS(f'Removed old backup: {old_backup}')
                    )

            # After cleanup, verify all remaining backups
            # This calls the verify_backup command for each backup file
            for backup_file in backup_files[:max_backups]:
                self.stdout.write(f"Verifying backup: {backup_file}")
                call_command('verify_backup', backup_file=str(backup_file))

        except Exception as e:
            logger.error(f"Backup management failed: {str(e)}")
            self.stdout.write(
                self.style.ERROR(f'Backup management failed: {str(e)}')
            )

3. Backup Verification

The verification process includes:

• Table structure comparison

• Row count validation

• Foreign key constraint checks

• Test restoration to temporary database

• Comprehensive integrity checks

  apps/realtime_timer/management/commands/verify_backup.py

    import os
    import sqlite3
    import tempfile
    from django.core.management.base import BaseCommand
    from django.conf import settings
    from django.apps import apps
    import logging
    from pathlib import Path
  
    logger = logging.getLogger(__name__)
  
    class Command(BaseCommand):
        """
        Django management command to verify the integrity of SQLite database backups.
        This command:
        1. Creates a temporary database
        2. Restores the backup to this temporary database
        3. Verifies:
           - Table structures match the original
           - Row counts match
           - Foreign key constraints are valid
  
        Usage:
            python manage.py verify_backup  # verifies latest backup
            python manage.py verify_backup --backup-file=/path/to/backup.db  # verifies specific backup
        """
  
        help = 'Verifies SQLite backup integrity and tests restoration'
  
        def add_arguments(self, parser):
            # Allow specifying a particular backup file to verify
            parser.add_argument(
                '--backup-file',
                help='Specific backup file to verify. If not provided, verifies latest backup.'
            )
  
        def verify_table_structure(self, source_conn, test_conn, table_name):
            """
            Compare table schemas between original and restored databases.
            Uses SQLite's sqlite_master table to get the CREATE TABLE statements
            and compares them.
            """
            source_schema = source_conn.execute("SELECT sql FROM sqlite_master WHERE type='table' AND name=?", (table_name,)).fetchone()
            test_schema = test_conn.execute("SELECT sql FROM sqlite_master WHERE type='table' AND name=?", (table_name,)).fetchone()
            return source_schema == test_schema
  
        def verify_row_counts(self, source_conn, test_conn, table_name):
            """
            Compare the number of rows in each table between original and restored databases.
            A simple COUNT(*) should match if the backup is valid.
            """
            source_count = source_conn.execute(f"SELECT COUNT(*) FROM {table_name}").fetchone()[0]
            test_count = test_conn.execute(f"SELECT COUNT(*) FROM {table_name}").fetchone()[0]
            return source_count == test_count
  
        def verify_foreign_keys(self, conn):
            """
            Verify that all foreign key constraints are valid in the restored database.
            Uses SQLite's PRAGMA foreign_key_check which returns empty list if all constraints are valid.
            """
            return conn.execute("PRAGMA foreign_key_check").fetchall() == []
  
        def handle(self, *args, **options):
            try:
                backup_dir = settings.SQLITE_BACKUP['BACKUP_DIRECTORY']
  
                # Determine which backup file to verify
                if options['backup_file']:
                    backup_path = Path(options['backup_file'])
                    if not backup_path.exists():
                        raise FileNotFoundError(f"Backup file not found: {backup_path}")
                else:
                    # If no specific file provided, get the most recent backup
                    backup_files = sorted(
                        Path(backup_dir).glob('backup_*.db'),
                        key=lambda x: x.stat().st_mtime,
                        reverse=True
                    )
                    if not backup_files:
                        raise FileNotFoundError("No backup files found")
                    backup_path = backup_files[0]
  
                # Create a temporary database file for testing restoration
                with tempfile.NamedTemporaryFile(suffix='.db', delete=False) as temp_db:
                    temp_db_path = temp_db.name
  
                try:
                    # Connect to both backup and temporary database
                    backup_conn = sqlite3.connect(str(backup_path))
                    test_conn = sqlite3.connect(temp_db_path)
  
                    # Restore the backup to temporary database
                    with backup_conn, test_conn:
                        backup_conn.backup(test_conn)
  
                    # Get all table names from Django models
                    django_tables = [
                        model._meta.db_table
                        for model in apps.get_models()
                    ]
  
                    # Store verification results
                    verification_results = {
                        'structure': [],  # Table structure verification results
                        'row_counts': [], # Row count verification results
                        'foreign_keys': True  # Foreign key constraint verification
                    }
  
                    # Verify each table
                    for table in django_tables:
                        # Check if table structures match
                        structure_match = self.verify_table_structure(backup_conn, test_conn, table)
                        verification_results['structure'].append({
                            'table': table,
                            'matches': structure_match
                        })
  
                        # Check if row counts match
                        count_match = self.verify_row_counts(backup_conn, test_conn, table)
                        verification_results['row_counts'].append({
                            'table': table,
                            'matches': count_match
                        })
  
                    # Verify foreign key constraints
                    verification_results['foreign_keys'] = self.verify_foreign_keys(test_conn)
  
                    # Check if all verifications passed
                    all_passed = (
                        all(r['matches'] for r in verification_results['structure']) and
                        all(r['matches'] for r in verification_results['row_counts']) and
                        verification_results['foreign_keys']
                    )
  
                    if all_passed:
                        # All checks passed - backup is valid
                        logger.info(f"Backup verification successful for {backup_path}")
                        self.stdout.write(
                            self.style.SUCCESS(f'Backup verification successful for {backup_path}')
                        )
                    else:
                        # Collect all failed checks for detailed error reporting
                        failed_checks = []
                        for result in verification_results['structure']:
                            if not result['matches']:
                                failed_checks.append(f"Structure mismatch in table {result['table']}")
  
                        for result in verification_results['row_counts']:
                            if not result['matches']:
                                failed_checks.append(f"Row count mismatch in table {result['table']}")
  
                        if not verification_results['foreign_keys']:
                            failed_checks.append("Foreign key constraints validation failed")
  
                        logger.error(f"Backup verification failed for {backup_path}: {', '.join(failed_checks)}")
                        self.stdout.write(
                            self.style.ERROR(f'Backup verification failed: {", ".join(failed_checks)}')
                        )
  
                finally:
                    # Clean up: remove temporary database
                    if os.path.exists(temp_db_path):
                        os.unlink(temp_db_path)
  
            except Exception as e:
                logger.error(f"Backup verification failed: {str(e)}")
                self.stdout.write(
                    self.style.ERROR(f'Backup verification failed: {str(e)}')
                )
  

4. GitHub Integration

• Automatic commits with readable timestamps

• Configurable Git user settings

• Push to dedicated backup branch

• Error handling and logging

apps/realtime_timer/management/commands/backup_git_push.py

import os
import subprocess
from datetime import datetime
from django.core.management.base import BaseCommand
from django.conf import settings
import logging

logger = logging.getLogger(__name__)

class Command(BaseCommand):
    """
    Django management command to commit and push database backups to GitHub.
    This command:
    1. Stages all new backup files
    2. Creates a commit with timestamp
    3. Pushes to the configured remote branch

    Usage:
        python manage.py backup_git_push

    Required Environment Variables:
        GIT_USER_NAME: Git user name for commits
        GIT_USER_EMAIL: Git user email for commits
        GIT_BRANCH: Branch to push backups (default: backup)
    """

    help = 'Commits and pushes database backups to GitHub'

    def add_arguments(self, parser):
        parser.add_argument(
            '--force',
            action='store_true',
            help='Skip confirmation prompt and force push backup',
        )

    def execute_git_command(self, command, error_message):
        """Execute git command and handle errors"""
        try:
            result = subprocess.run(
                command,
                cwd=settings.BASE_DIR,
                check=True,
                capture_output=True,
                text=True
            )
            return result.stdout.strip()
        except subprocess.CalledProcessError as e:
            logger.error(f"{error_message}: {e.stderr}")
            raise

    def handle(self, *args, **options):
        try:
            backup_dir = settings.SQLITE_BACKUP['BACKUP_DIRECTORY']
            git_user = settings.GIT_USER_NAME
            git_email = settings.GIT_USER_EMAIL

            if not all([git_user, git_email]):
                raise ValueError("GIT_USER_NAME and GIT_USER_EMAIL must be set in environment")

            # Configure git user for this commit
            self.execute_git_command(
                ['git', 'config', 'user.name', git_user],
                "Failed to set git user name"
            )
            self.execute_git_command(
                ['git', 'config', 'user.email', git_email],
                "Failed to set git user email"
            )

            # Add confirmation prompt unless --force flag is used
            if not options['force']:
                confirm = input(
                    "\nWARNING: You are about to push database backups to a remote repository.\n"
                    "This may override existing data in the repository.\n"
                    "Are you sure you want to continue? [y/N]: "
                ).lower()
                if confirm not in ['y', 'yes']:
                    self.stdout.write(self.style.WARNING('Backup push cancelled'))
                    return

            # Stage all backup files
            backup_pattern = os.path.join(backup_dir, 'backup_*.db')
            self.execute_git_command(
                ['git', 'add', backup_pattern],
                "Failed to stage backup files"
            )

            # Create commit with timestamp
            # this time is in UTC+0 
            timestamp = datetime.now().strftime('%B %d, %Y %I:%M %p')
            commit_message = f"Database backup - {timestamp}"
            self.execute_git_command(
                ['git', 'commit', '-m', commit_message],
                "Failed to create commit"
            )

            # Push to remote
            self.execute_git_command(
                ['git', 'push', 'origin'],
                "Failed to push to GitHub"
            )

            logger.info("Successfully pushed backup to GitHub")
            self.stdout.write(
                self.style.SUCCESS('Successfully pushed backup to GitHub')
            )

        except Exception as e:
            logger.error(f"Failed to push backup to GitHub: {str(e)}")
            self.stdout.write(
                self.style.ERROR('Failed to push backup to GitHub')
            )

5. Centralized Management

A single command orchestrates:

• Backup creation

• Rotation management

• Integrity verification

• GitHub synchronization

apps/realtime_timer/management/commands/backup_manager.py

import logging
from django.core.management.base import BaseCommand
from django.core.management import call_command
from datetime import datetime

logger = logging.getLogger(__name__)

class Command(BaseCommand):
    """
    Django management command to handle all backup-related operations:
    1. Creates a new backup
    2. Manages backup retention (removes old backups)
    3. Verifies backup integrity
    4. Pushes backups to GitHub

    Usage:
        python manage.py backup_manager
    """

    help = 'Manages all backup operations (create, verify, rotate, and push to GitHub)'

    def handle(self, *args, **options):
        try:
            timestamp = datetime.now().strftime('%B %d, %Y %I:%M %p')
            self.stdout.write(f"Starting backup operations at {timestamp}",style_func=self.style.WARNING)

            # Step 1: Create new backup
            self.stdout.write("Creating new backup...",style_func=self.style.WARNING)
            call_command('backup_database')

            # Step 2: Manage backups (includes verification)
            self.stdout.write("Managing and verifying backups...",style_func=self.style.WARNING)
            call_command('manage_backups')

            # Step 3: Push to GitHub
            self.stdout.write("Pushing backups to GitHub...",style_func=self.style.WARNING)
            call_command('backup_git_push')

            self.stdout.write(
                self.style.SUCCESS('All backup operations completed successfully')
            )

        except Exception as e:
            logger.error(f"Backup operations failed: {str(e)}")
            self.stdout.write(
                self.style.ERROR(f'Backup operations failed: {str(e)}')
            )

Configuration

Settings in Django

# SQLite backup settings
SQLITE_BACKUP = {
    'BACKUP_DIRECTORY': os.path.join(BASE_DIR, 'backups'),
    'BACKUP_MAX_COUNT': 10,  # Maximum number of backups to keep
}

GIT_USER_NAME = os.environ.get('GIT_USER_NAME')
GIT_USER_EMAIL = os.environ.get('GIT_USER_EMAIL')

• GIT_USER_NAME: Git username for commits

• GIT_USER_EMAIL: Git email for commits

Automation

Cron Job Setup

Run backups every 6 hours:

0 */6 * * * cd /home/normal_user/focus-timer-v/ && . /home/normal_user/focus-timer-v/.env && /home/normal_user/focus-timer-v/venv/bin/python manage.py backup_manager >> /home/normal_user/focus-timer-v/logs/crontab_logs.txt 2>&1; echo "Cron ran at $(date)" >> /home/normal_user/focus-timer-v/logs/debug_cron.log

Here is an screenshot showing it in action 👇🏻

Conclusion

This backup system provides a robust, automated solution for SQLite database backups in Django applications. It ensures data safety through multiple verification layers and maintains backup integrity while requiring minimal manual intervention.

The Big Picture: How Timer Ticks

The Tymr focus timer application uses a unique combination of Redis, Django Channels, and client-side JavaScript to manage real-time timer updates. At its core, a Redis Scheduler continuously checks for scheduled cycle changes. When a change is due, it triggers an update in the Django application, which then uses Django Channels to push this update to connected WebSocket clients. On the client side, a combination of JavaScript and Django template language renders the timer interface and handles user interactions. These interactions are sent back to the Django application via WebSocket, which may result in new cycle schedules being added to Redis. All persistent data, including user information and session details, is stored in a SQLite3 database. This architecture allows for precise, server-side timer management while providing a responsive, real-time experience for users.

Here's a diagram illustrating this architecture:

This diagram shows the flow of data and interactions between the different components of the Tymr application, highlighting the central role of the Redis Scheduler and Django Channels Consumer in managing the focus timer sessions.

Your browser talks to Nginx, our trusty web server.

1. Nginx routes HTTP requests to Django and WebSocket connections to Uvicorn (our ASGI server).

2. Both Django and Uvicorn interact with our SQLite3 database (yep, keeping it simple!) and Redis.

3. Our Redis Scheduler keeps an eye on when to change the cycle. we just add the cycle details along with focus session id to the redis zset & in the redis_scheduler.py, we keep on scanning the redis zset if there is any data and we process them if found.

4. OneSignal handles push notifications when your focus session ends.

5. when a session start, consumer send the timer update to clients & client browsers use web worker to keep the timer ticking. one there is an update about next cycle, that also sent to client browser using websocket connection.

Now, let's zoom in on some of the cool parts!

the decision to use sqlite3 was to keep things simple. i don’t think i need postgresql for this so far. also i saw a short clip by dhh inspiring me for simplicity in the start and not caring much about scalability because we are not there yet. although i worked on this project mainly for learning purpose & experimenting different things i learn.

Battling Browser Naps with Web Workers

Ever noticed how sometimes your browser tabs seem to doze off when you're not looking at them? This can mess with JavaScript timers, which is not great for a focus timer app. To combat this, we're using Web Workers. These little champions run in the background, keeping our timer ticking even when the browser tab is sleeping.

Here's a snippet of how we set this up:

// In focus_session.js
const worker = new Worker('hack_timer_worker.js');

worker.onmessage = function(event) {
    // Handle timer updates
};

// Start a timer
worker.postMessage({
    name: 'setInterval',
    time: 1000  // Update every second
});

This keeps our timer accurate, even if you switch tabs or minimize your browser. Pretty neat, huh? but to make it more simple, i just add this script before loading the timer related code. https://github.com/turuslan/HackTimer which solves most of tab sleep issues.

Redis Scheduler: The Time Lord of Our App

Now, let's talk about our Redis Scheduler. This is the behind-the-scenes magician that makes sure your focus cycles change at the right time, even if you close your browser entirely.

Here's how it works:

1. When you start a session, we schedule the next cycle change in Redis using a sorted set (zset).

2. Our Redis Scheduler (running as a Django management command) constantly checks for due changes.

3. When it's time, it triggers the cycle change and schedules the next one.

Let's look at some code:

# In redis_scheduler.py
class RedisScheduler:
    async def run(self):
        while True:
            now = time.time()
            due_changes = await self.redis.zrangebyscore("scheduled_cycle_changes", 0, now)
            for session_id in due_changes:
                await self.process_change(session_id)
            await asyncio.sleep(1)

    async def process_change(self, session_id):
        session = await selectors.get_session_by_id_async(session_id)
        timer_service = AsyncTimerService(session_id, session.owner, session.owner.username)
        await timer_service.change_cycle_if_needed(session)
        await self.redis.zrem("scheduled_cycle_changes", session_id)
        await timer_service.schedule_next_cycle_change(self.redis)

This setup allows for precise timing without relying solely on client-side JavaScript. Cool, right? well here is the complete redis_scheduler.py file incase you wanna know.

import asyncio
import redis.asyncio as redis
from django.conf import settings
from channels.layers import get_channel_layer
import logging
import time
from django.core.management.base import BaseCommand
from apps.realtime_timer.business_logic import selectors
from apps.realtime_timer.business_logic.services import AsyncTimerService
from apps.realtime_timer.models import FocusSession

logger = logging.getLogger(__name__)

class RedisScheduler:
    def __init__(self):
        self.redis = redis.Redis.from_url(f"redis://{settings.REDIS_HOST}:{settings.REDIS_PORT}")
        self.channel_layer = get_channel_layer()

    async def run(self):
        while True:
            try:
                now = time.time()
                due_changes = await self.redis.zrangebyscore("scheduled_cycle_changes", 0, now, start=0, num=100)
                if due_changes:
                    for session_id in due_changes:
                        logger.info("--------------------------------------------")
                        await self.process_change(session_id)
                        logger.info("--------------------------------------------")
                    # TODO: remove this wait if possible
                    await asyncio.sleep(0.1)
                else:
                    spinner = "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏"
                    for _ in range(10):
                        print(f"\rWaiting for changes {spinner[_ % len(spinner)]}", end="", flush=True)
                        await asyncio.sleep(0.1)
                    print("\r" + " " * 30 + "\r", end="", flush=True)
                    await asyncio.sleep(1)
            except redis.RedisError as e:
                logger.error(f"Redis error: {e}")
                await asyncio.sleep(5)
            except Exception as e:
                logger.error(f"Unexpected error: {e}")
                await asyncio.sleep(5)

    async def process_change(self, session_id):
        try:
            session_id = session_id.decode()
            session = await selectors.get_session_by_id_async(session_id)
            session_owner = await selectors.get_session_owner_async(session)
            if not session:
                logger.error(f"Session {session_id} not found, skipping processing")
                await self.redis.zrem("scheduled_cycle_changes", session_id)
                return
            timer_service = AsyncTimerService(session_id, session_owner, session_owner.username)
            if session.timer_state == FocusSession.TIMER_RUNNING:
                logger.info(f"calling timer_service.change_cycle_if_needed for session {session_id}")
                await timer_service.change_cycle_if_needed(session)
                logger.info(f"calling timer_service.schedule_next_cycle_change for session {session_id}")
                await self.redis.zrem("scheduled_cycle_changes", session_id)
                await timer_service.schedule_next_cycle_change(self.redis)
                logger.info(f"cycle change completed for session {session_id}")
            else:
                logger.info(f"session {session_id} is not running, skipping cycle change")
                await self.redis.zrem("scheduled_cycle_changes", session_id)
        except Exception as e:
            await self.redis.zrem("scheduled_cycle_changes", session_id)
            logger.error(f"Error processing change for session {session_id}: {e}")
            logger.exception("Full traceback:")


class Command(BaseCommand):
    help = "Runs the Redis scheduler for focus session cycle changes"

    def handle(self, *args, **options):
        scheduler = RedisScheduler()
        asyncio.run(scheduler.run())

Race Conditions: When Timing is Everything

Now, let's talk about a tricky problem in concurrent programming: race conditions. Imagine two people trying to update the same focus session at the same time. Chaos, right?

We faced this issue with our focus cycles. Multiple processes could try to update the same session simultaneously, leading to:

• Incorrect cycle transitions

• Wrong focus period calculations

• Inconsistent timer states

To tackle this, we use Django's select_for_update(). It's like putting a "Do Not Disturb" sign on our database rows. Here's how we use it:

@database_sync_to_async
def get_current_cycle_locked_async(session: FocusSession):
    return FocusCycle.objects.select_for_update().get(id=session.current_cycle_id)

But be careful! Using select_for_update() can lead to deadlocks if not used properly. Here are some precautions:

1. Always acquire locks in the same order across your application.

2. Keep the locked section as short as possible.

3. Use timeouts to prevent indefinite waiting.

For example:

from django.db import transaction

@database_sync_to_async
def update_session(session_id):
    with transaction.atomic():
        session = FocusSession.objects.select_for_update(nowait=True).get(id=session_id)
        # Perform updates here
        session.save()

The nowait=True option helps prevent deadlocks by raising an error immediately if the lock can't be acquired and you can catch that error and handle those cases manually.

Async Consumers: Real-Time Magic

Last but not least, let's chat about our async consumers. These are the multitasking ninjas of our application, handling real-time communications without breaking a sweat.

We use Django Channels to handle WebSocket connections asynchronously. This allows us to manage multiple user connections simultaneously. Here's a simplified version of our consumer:

class FocusSessionConsumer(AsyncWebsocketConsumer):
    async def connect(self):
        self.session_id = self.scope["url_route"]["kwargs"]["session_id"]
        self.session_group_name = f"focus_session_{self.session_id}"
        await self.channel_layer.group_add(self.session_group_name, self.channel_name)
        await self.accept()

    async def receive(self, text_data):
        data = json.loads(text_data)
        if data["action"] == "toggle_timer":
            await self.toggle_timer()
        # ................................

    @check_session_owner_async
    async def toggle_timer(self):
        timer_state = await self.timer_service.toggle_timer()
        if timer_state == "paused":
            await self.timer_service.cancel_scheduled_cycle_change_if_timer_stopped(self.redis_client, self.session_id)
        elif timer_state == "resumed":
            await self.timer_service.schedule_next_cycle_change(redis_client=self.redis_client)

    # rest of the logic...................

This setup allows us to handle real-time updates for multiple users smoothly!

Wrapping Up

Building Tymr was like assembling a high-tech clock with a sprinkle of cloud magic. From battling browser naps with Web Workers to orchestrating time with our Redis Scheduler, and managing real-time connections with async consumers - each component plays a crucial role in keeping everything ticking smoothly.

I deployed this django app to https://tymr.online so incase you want to learn about deployment of Django app on vps then check this link

https://selftaughtdev.hashnode.dev/comprehensive-django-deployment-guide-for-beginners

Got any questions or want to dive deeper into any part? Drop a comment below - I'd love to geek out about it with you!

p.s. I am still learning & i may have missed some points here or not explain them well.

1. Introduction

2. Prerequisites

3. Setting Up Your VPS

4. Securing Your Server

5. Installing Required Software

6. Configuring Your Django Project

7. Setting Up Nginx and Uvicorn

8. Configuring SSL with Let's Encrypt

9. Setting Up Supervisor

10. Firewall Configuration

11. Deploying Your Django Application

12. Running the Redis Scheduler

13. Monitoring and Maintenance

14. Troubleshooting

15. Conclusion

1. Introduction

This guide will walk you through the process of deploying a Django application with ASGI (Uvicorn) to a Virtual Private Server (VPS). We'll also cover setting up a Redis scheduler, configuring Nginx as a reverse proxy, securing your server with a firewall, and enabling HTTPS. This guide is designed for beginners, so we'll explain each step in detail.

2. Prerequisites

Before we begin, make sure you have:

• A VPS running Ubuntu 22.04

• A domain name (e.g., tymr.online) pointed to your VPS's IP address

• Your Django project code (including requirements.txt)

• Basic familiarity with command-line interfaces

3. Setting Up Your VPS

First, we need to access our VPS. We'll use SSH (Secure Shell) for this.

ssh root@your_server_ip

Replace your_server_ip with your VPS's IP address. You'll be prompted for a password, which you should have received from your VPS provider or in most VPS you can directly login into server after pasting the local desktop’s ssh public key into the server during the setup process. for example, in digital ocean you have this feature here while creating a droplet ( VPS ) 👇

Once logged in, update your system:

sudo apt update
sudo apt upgrade -y

These commands update the list of available packages and upgrade them to their latest versions.

4. Securing Your Server

Creating a New User

It's best practice to avoid using the root user. Let's create a new user:

adduser username

Replace username with your desired username. You'll be prompted to set a password and provide some optional information.

Grant this user sudo privileges:

usermod -aG sudo username

This command adds the user to the sudo group, allowing them to run commands with superuser privileges.

Setting Up SSH Key Authentication

SSH keys are more secure than passwords. On your local machine, generate an SSH key pair:

ssh-keygen -t rsa -b 4096

This creates a public/private key pair. The public key can be freely shared, while the private key must be kept secret.

Copy the public key to your server:

ssh-copy-id username@your_server_ip

Now, let's disable password authentication for SSH. On the server:

sudo nano /etc/ssh/sshd_config

Find and modify these lines:

PasswordAuthentication no
PermitRootLogin no

Save the file (Ctrl+X, then Y, then Enter) and restart the SSH service:

sudo systemctl restart sshd

Now login as non root user you just created.

5. Installing Required Software

Let's install the necessary software:

sudo apt install build-essential python3-dev python3.11 python3.11-venv python3-pip nginx redis

This installs Python 3.11, pip (Python package manager), and Nginx (web server).

6. Configuring Your Django Project

Create a directory for your project:

mkdir ~/myproject
cd ~/myproject

Upload your project files to this directory. You can use SCP, SFTP, or Git for this.

Create a virtual environment:

python3.11 -m venv venv
source venv/bin/activate

Install your project dependencies:

pip install -r requirements.txt

Also, install Uvicorn:

pip install uvicorn

Understanding and Setting Up User Permissions for our Django project

When deploying a Django application, proper file permissions and ownership are crucial for security and functionality. Here's a detailed guide tailored for beginners:

Understanding Users and Groups

In our setup, we'll work with three main entities:

1. Your user account (e.g., 'normaluser')

2. The web server user (usually 'www-data' for Nginx on Ubuntu)

3. The root user

Best Practices for Django Project Files

1. Project Directory Ownership: Your user should own the project directory and most files.

    sudo chown -R normaluser:normaluser /home/normaluser/myproject
   

2. Project Directory Permissions: Set 755 (drwxr-xr-x) for directories and 644 (-rw-r--r--) for files.

    find /home/normaluser/myproject -type d -exec chmod 755 {} \;
    find /home/normaluser/myproject -type f -exec chmod 644 {} \;
   

3. Sensitive Files: For files like settings.py, use more restrictive permissions.

    chmod 600 /home/normaluser/myproject/myproject/settings.py
   

4. Static and Media Directories: The web server needs read access to these.

    sudo chown -R normaluser:www-data /home/normaluser/myproject/static
    sudo chown -R normaluser:www-data /home/normaluser/myproject/media
    sudo chmod -R 750 /home/normaluser/myproject/static
    sudo chmod -R 750 /home/normaluser/myproject/media
   

5. Log Directory: Create a logs directory within your project and set appropriate permissions.

    mkdir -p /home/normaluser/myproject/logs
    sudo chown -R normaluser:www-data /home/normaluser/myproject/logs
    sudo chmod -R 770 /home/normaluser/myproject/logs
   

6. SQLite Database (if used): If you're using SQLite, secure the database file.

    sudo chown normaluser:www-data /home/normaluser/myproject/db.sqlite3
    sudo chmod 660 /home/normaluser/myproject/db.sqlite3
   

7. Virtual Environment: Keep your virtual environment secure.

    chmod -R 750 /home/normaluser/myproject/venv
   

Explanation of Permissions

• 755 for directories: Owner can read/write/execute, others can read/execute.

• 644 for regular files: Owner can read/write, others can read.

• 600 for sensitive files: Only owner can read/write.

• 750 for static/media: Owner can read/write/execute, group (www-data) can read/execute.

• 770 for logs: Owner and group can read/write/execute.

• 660 for SQLite: Owner and group can read/write.

Security Best Practices

1. Principle of Least Privilege: Give only the permissions necessary for each file and directory.

2. Use a Non-Root User: Always run your Django application as a non-root user.

3. Separate Media Uploads: If possible, store user-uploaded files outside of the project directory.

4. Regular Audits: Periodically check and update file permissions.

5. Use Environment Variables: Store sensitive information like secret keys in environment variables, not in files.

6. Secure the Secret Key: Ensure Django's SECRET_KEY is not in version control and has restricted access.

7. Backup Permissions: When backing up, preserve file permissions.

Applying Changes

After setting permissions, restart your web server and Django application:

sudo systemctl restart nginx
sudo supervisorctl restart all

Verifying Permissions

You can verify permissions using the ls -l command. For example:

ls -l /home/normaluser/myproject

This will show you the permissions for files and directories in your project root.

Remember, these settings are a starting point. You may need to adjust based on your specific setup and security requirements. Always test thoroughly after changing permissions to ensure your application functions correctly.

7. Nginx Configuration

Before we set up our Nginx configuration, let's remove the default configuration:

sudo rm /etc/nginx/sites-enabled/default

Now, let's create our configuration file:

sudo nano /etc/nginx/sites-available/myproject

Add the following content:

server {
    # replace your domain names with placeholder tymr.online
    server_name tymr.online www.tymr.online;  # Your domain name(s)

    location = /favicon.ico { 
        access_log off; 
        log_not_found off; 
    }  # Efficiently handle favicon requests

    location /static/ {
        root /home/username/myproject;  # Path to your static files
    }  # Serve static files directly

    location / {
        proxy_pass http://unix:/home/username/myproject/myproject.sock;  # Pass requests to Uvicorn
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }  # Proxy settings for Uvicorn
}

Let's break this down:

• server_name: Specifies which domain names this server block should respond to.

• location = /favicon.ico: Efficiently handles favicon requests.

• location /static/: Configures Nginx to serve static files directly, without passing the request to Django.

• location /: The main block that handles all other requests.

  • proxy_pass: Forwards requests to our Uvicorn server.

  • The proxy_set_header directives: Pass necessary headers to the Django application.

After creating this file, enable it:

sudo ln -s /etc/nginx/sites-available/myproject /etc/nginx/sites-enabled

This creates a symbolic link, effectively enabling our configuration.

Test the Nginx configuration:

sudo nginx -t

If it's okay, restart Nginx:

sudo systemctl restart nginx

8. Setting Up SSL with Let's Encrypt

We'll use Snap to install Certbot, which automates the process of obtaining and renewing Let's Encrypt certificates.

1. Ensure Snap is up-to-date:

    sudo snap install core; sudo snap refresh core
   

   This installs and updates the core snap package.

2. Remove any old Certbot installations:

    sudo apt remove certbot
   

3. Install Certbot via Snap:

    sudo snap install --classic certbot
   

   The --classic flag allows Certbot to access system resources outside of the snap's confined environment.

4. Create a symbolic link for easy access:

    sudo ln -s /snap/bin/certbot /usr/bin/certbot
   

5. Obtain and install the certificate:

    sudo certbot --nginx -d tymr.online -d www.tymr.online
   

   This command will:

   • Obtain a certificate for your domains

   • Automatically configure Nginx to use the new certificate

   • Set up automatic renewal

Follow the prompts during the Certbot execution. It will ask for your email and agreement to terms of service.

For more detailed information, you can refer to this similar guide: How To Secure Nginx with Let's Encrypt on Ubuntu 22.04

9. Configuring Supervisor

Supervisor is a process control system that allows you to monitor and control a number of processes on UNIX-like operating systems. We use Supervisor because:

1. It automatically restarts processes if they crash.

2. It starts processes on system boot.

3. It provides an easy way to start, stop, and monitor processes.

4. It logs stdout and stderr of the processes it manages.

Here's how to set it up:

1. Install Supervisor:

    sudo apt install supervisor
   

2. Create a configuration file:

    sudo nano /etc/supervisor/conf.d/myproject.conf
   

3. Add the following content:

[program:myproject]
command=/home/username/myproject/venv/bin/uvicorn myproject.asgi:application --unix-socket /home/username/myproject/myproject.sock
directory=/home/username/myproject
user=username
autostart=true
autorestart=true
stderr_logfile=/var/log/myproject.err.log
stdout_logfile=/var/log/myproject.out.log

[program:celery]
command=/home/username/myproject/venv/bin/celery -A myproject worker -l info
directory=/home/username/myproject
user=username
autostart=true
autorestart=true
stderr_logfile=/var/log/celery.err.log
stdout_logfile=/var/log/celery.out.log

[program:redis_scheduler]
command=/home/username/myproject/venv/bin/python manage.py redis_scheduler
directory=/home/username/myproject
user=username
autostart=true
autorestart=true
stderr_logfile=/var/log/redis_scheduler.err.log
stdout_logfile=/var/log/redis_scheduler.out.log

This configuration sets up three processes:

• Your Django application running with Uvicorn

• A Celery worker for background tasks

• Your custom Redis scheduler

4. Reload Supervisor to apply changes:

    sudo supervisorctl reread
    sudo supervisorctl update
   

Supervisor is different from simple systemd services because it provides more fine-grained control and is specifically designed for managing application processes, making it ideal for complex setups like Django projects with multiple components.

10. Firewall Configuration

UFW (Uncomplicated Firewall) is a user-friendly interface for managing iptables, the default firewall management tool for Ubuntu.

Enable UFW:

sudo ufw enable

Allow necessary connections:

sudo ufw allow ssh
sudo ufw allow 'Nginx Full'

These commands allow SSH connections and both HTTP and HTTPS traffic for Nginx.

Check the status:

sudo ufw status

Final List of UFW Commands:

sudo ufw allow OpenSSH # You need to allow SSH so you can access your VPS remotely.
sudo ufw allow 'Nginx HTTP'
sudo ufw allow 'Nginx HTTPS'
sudo ufw allow 8000  # Optional, for development server
sudo ufw enable
sudo ufw status

11. Deploying Your Django Application

Collect static files:

python manage.py collectstatic

Apply migrations:

python manage.py migrate

12. Running the Redis Scheduler

The Redis scheduler should now be managed by Supervisor. You can check its status with:

sudo supervisorctl status

13. Custom Scripts for Process Management

To simplify starting and stopping all processes, you can create custom scripts. Here are examples:

1. Create a start script:

    nano ~/start_all.sh
   

   Add the following content:

    #!/bin/bash
    sudo supervisorctl start all
    sudo systemctl start nginx
   

2. Create a stop script:

    nano ~/stop_all.sh
   

   Add the following content:

    #!/bin/bash
    sudo supervisorctl stop all
    sudo systemctl stop nginx
   

3. Make these scripts executable:

    chmod +x ~/start_all.sh ~/stop_all.sh
   

Now you can start all processes with ./start_all.sh and stop them with ./stop_all.sh.

Remember to run these scripts with sudo if your user doesn't have the necessary permissions.

14. Monitoring and Maintenance

Regularly update your system:

sudo apt update
sudo apt upgrade

Monitor your logs:

tail -f /var/log/myproject.err.log
tail -f /var/log/myproject.out.log
tail -f /var/log/redis_scheduler.err.log
tail -f /var/log/redis_scheduler.out.log

15. Troubleshooting

• If your site isn't accessible, check Nginx status: sudo systemctl status nginx

• If Uvicorn isn't running, check Supervisor: sudo supervisorctl status

• Check Nginx error logs: sudo tail -f /var/log/nginx/error.log

• Ensure your firewall isn't blocking connections: sudo ufw status

16. Conclusion

You've now deployed your Django application with Uvicorn, set up Nginx as a reverse proxy, configured SSL, and set up a Redis scheduler. Your application should be accessible via HTTPS at your domain.

Remember to keep your system and applications updated, monitor your logs, and maintain good security practices.

but first, here is a screencast of how it looks like in action when i implemented it my project. i have replace the project code with some generic example code because i can’t share the project code as it’s confidential.

The Challenge: your company have a blog management system that required displaying a table of blog posts with various attributes. they need server-side processing to handle potentially thousands of posts efficiently, with dynamic columns based on user permissions, and complex sorting and filtering capabilities.

The Solution: Let's break this down step-by-step:

1. Setting Up the Frontend: First, I set up DataTables on the client side:

// This setup enables server-side processing 
$(document).ready(function () {
  let host_url = window.location.origin;
  var dataTable = $('#blog-posts-table').DataTable({
    serverSide: true,
    sAjaxSource: `${host_url}/api/blog-posts-data`,
    paging: true,
    processing: true,
    // disables ordering on the last two columns (action buttons).
    columnDefs: [
      { orderable: false, targets: -1 },
      { orderable: false, targets: -2 }
    ]
  });
});

2. Django View Magic: The real complexity lay in the Django view. Here's the complete view.

@login_required
@user_passes_test(lambda u: u.has_perm("view_blog_posts"))
def blog_posts_datatables_api_view(request):
    request_data = request.GET.copy()
    # we need to return the value of draw in int format
    # it is strongly recommended for security reasons that 
    # you cast this parameter to an integer, rather than simply 
    # echoing back to the client what it sent in the draw parameter, 
    # in order to prevent Cross Site Scripting (XSS) attacks
    request_data["draw"] = int(request_data.get("draw", 1))
    user = request.user
    blog = Blog.objects.for_user(user)
    approved_categories = get_approved_categories(request, blog)

    post_columns = _get_post_columns(blog)
    order_by_list = _get_col_list_for_ordering(request.GET, post_columns)

    search_query = request.GET.get("sSearch")
    entries_per_page = int(request.GET.get("iDisplayLength", 10))
    entries_start = int(request.GET.get("iDisplayStart", 0))

    search_db_query = Q()
    if search_query:
        search_db_query = (
            Q(author__first_name__icontains=search_query)
            | Q(author__last_name__icontains=search_query)
            | Q(title__icontains=search_query)
            | Q(content__icontains=search_query)
        )

    posts = (
        blog.blogpost_set.filter(
            deleted_on__isnull=True,
            draft=False,
            author__is_active=True,
            category__in=approved_categories,
        )
        .select_related("author", "category")
        .order_by(*order_by_list)
    )

    request_data["recordsTotal"] = posts.count()

    if search_query:
        posts = posts.filter(search_db_query)

    request_data["recordsFiltered"] = posts.count()

    posts = posts[entries_start : entries_start + entries_per_page]

    data = []
    for post in posts:
        _post_data = []
        _author_link = ""
        if blog.show_author_id:
            if post.author.id:
                _author_link = f'<a href="{post.author.get_absolute_url()}">{post.author.id}</a>'
                _post_data.append(_author_link)
                _post_data.append(post.author.get_full_name())

        if not _author_link:
            _post_data.append(f'<a href="{post.author.get_absolute_url()}">{post.author.get_full_name()}</a>')

        _post_data.append(post.title)
        _post_data.append(post.category.__str__())
        _post_data.append(post.published_date.strftime("%Y-%m-%d %H:%M:%S"))
        _post_data.append(post.view_count)
        _post_data.append(yesno(post.comments_enabled, "Enabled,Disabled"))

        edit_url = reverse(
            "blog:post-update",
            kwargs={"post_id": post.id},
        )
        _post_data.append(f'<a href="{edit_url}"><span class="material-symbols-outlined md-18">edit</span></a>')

        toggle_comments_url = reverse(
            "blog:toggle-post-comments",
            kwargs={"post_id": post.id},
        )
        comments_text = yesno(post.comments_enabled, "comments_disabled,comments")
        title = "Disable Comments" if post.comments_enabled else "Enable Comments"
        _post_data.append(
            f'<a href="{toggle_comments_url}"><span title="{title}" class="material-symbols-outlined">{comments_text}</span></a>'
        )

        data.append(_post_data)

    request_data["data"] = data
    return JsonResponse(request_data)

Key Points:

• We're copying the request data and modifying it throughout the view.

• Dynamic column handling based on blog settings.

• Complex filtering using Q objects.

• Efficient querying with select_related.

• Pagination handled server-side.

• Custom data formatting for each post, including HTML for action buttons.

Now there are few params sent in request by client & here is the info about them.

Here's the formatted table in Markdown:

there are other params also which you can read in detail here - https://datatables.net/manual/server-side#Returned-data

3. Helper Functions: I created helper functions to keep the main view clean:

def _get_post_columns(blog):
    post_columns = []
    if blog.show_author_id:
        post_columns.append("author__id")
    post_columns.extend([
        "author__first_name",
        "title",
        "category",
        "published_date",
        "view_count",
        "comments_enabled"
    ])
    return post_columns

def _get_col_list_for_ordering(request_data, post_columns):
    order_by_list = []
    for key, value in request_data.items():
        if key.startswith("iSortCol_"):
            col_index = value
            col_name_key = f"mDataProp_{col_index}"
            col_name = request_data.get(col_name_key)
            col_name = post_columns[int(col_name)]
            sort_dir_key = f'sSortDir_{key.split("_")[1]}'
            sort_dir = request_data.get(sort_dir_key, "asc")
            if sort_dir == "desc":
                col_name = f"-{col_name}"
            order_by_list.append(col_name)
    return order_by_list or [post_columns[0]]

This function creates a list of columns to use in the queryset's order_by() method:

• It iterates through the request data (which comes from DataTables).

• It looks for keys starting with "iSortCol_", which indicate sorting columns.

• For each sorting column:

  • It gets the column index and finds the corresponding column name.

  • It determines the sort direction (ascending or descending).

  • If descending, it prepends a minus sign to the column name.

  • It adds this to the order_by_list.

• If no sorting is specified, it defaults to the first column in post_columns.

This function effectively translates DataTables' sorting parameters into Django ORM ordering syntax, allowing for dynamic, multi-column sorting as specified by the user in the frontend.

4. Template Changes: In the HTML template, I set up the table structure:

<table id="blog-posts-table" class="table table-striped table-hover table-sm table-responsive">
  <thead>
    <tr>
      {% if blog.show_author_id %}
        <th>ID</th>
      {% endif %}
      <th>AUTHOR</th>
      <th>TITLE</th>
      <th>CATEGORY</th>
      <th>PUBLISHED DATE</th>
      <th>VIEWS</th>
      <th>COMMENTS</th>
      <th><span class="material-symbols-outlined md-18">edit</span></th>
      <th><span title="Toggle Comments" class="material-symbols-outlined">comments</span></th>
    </tr>
  </thead>
  <tbody></tbody>
</table>

5. URL Configuration: I added a new URL pattern for the API view:

path(
    "api/blog-posts-data",
    views.blog_posts_datatables_api_view,
    name="blog-posts-datatables-data",
),

The Result: The end product was a highly efficient and flexible blog post management table. It loads quickly, even with thousands of posts, and provides smooth sorting, searching, and pagination - all handled server-side.

Key Takeaways:

1. Copying and modifying the request data allows for flexible response handling.

2. Server-side processing is crucial for large datasets.

3. Dynamic column handling requires careful planning but offers great flexibility.

4. Efficient database querying (using select_related and filter) is key to maintaining performance.

5. Breaking down complex logic into helper functions improves readability and maintainability.

6. Handling HTML generation server-side for action buttons allows for dynamic permissions and URLs.

This solution has been a game-changer for my blog management project. It's scalable, efficient, and provides a great user experience while handling complex data structures and permissions.

Have you tackled similar challenges with large, dynamic datasets in Django? I'd love to hear your approaches and any tips you might have!

p.s - i am still learning and chances are there may be better approach than this to implement server side datatables. i just shared my way of doing this but if i found a new better way then i will mention that here.

Default Django logs are okay, but they lack juicy details like IP addresses, browsers, and user info. Wouldn't it be cool to have all that at your fingertips?

Enter Custom Logging Filters

Django lets us create custom filters to add extra info to our logs. Here's how:

1. First, set up your logging config in settings.py. It'll look something like this:

LOGGING = {
    # ... other config ...
    "filters": {
        "custom_request_filter": {
            "()": "path.to.your.CustomRequestFilter",
        },
    },
    # ... more config ...
}

2. Now, let's create our custom filter:

import logging
from django.core.exceptions import ObjectDoesNotExist


class CustomRequestFilter(logging.Filter):
    def filter(self, record):
        if hasattr(record, "request"):
            request = record.request  # type: ignore
            try:
                record.ip_address = request.META.get("REMOTE_ADDR", "-")
                record.device_type = request.META.get("HTTP_SEC_CH_UA_PLATFORM")
                # ex string - "Brave";v="129", "Not=A?Brand";v="8", "Chromium";v="129"
                record.browser = request.META.get("HTTP_SEC_CH_UA").split(";")[0]
            except AttributeError:
                record.browser = "-"
                record.device_type = "-"
                record.ip_address = "-"
            try:
                record.user = request.user.username if request.user.is_authenticated else "unauthenticated"
            except (ObjectDoesNotExist, AttributeError):
                record.user = "unauthenticated"
            record.http_status = getattr(record, "status_code", "-")
        else:
            record.ip_address = "-"
            record.user = "-"
            record.device_type = "-"
            record.browser = "-"
            record.http_status = "-"
        return True

3. Apply the filter to your handlers:

"handlers": {
    "file": {
        # ... other config ...
        "filters": ["custom_request_filter"],
    },
},

4. Update your formatter to use the new attributes:

"formatters": {
    "verbose": {
        "format": "{asctime} {ip_address} {user} {message}",
        "style": "{",
    },
},

Bonus: Logging in WebSocket Consumers

Don't forget about your WebSocket consumers! They need love too. Here's a quick tip:

class FocusSessionConsumer(AsyncWebsocketConsumer):
    async def connect(self):
        self.request = self._generate_request_metadata()
        # ... other connect logic ...
        logger.info(f"{self.user.username} connected", extra={"request": self.request})

    def _generate_request_metadata(self):
        request = HttpRequest()
        # Populate request.META with relevant info
        return request

Here are some code snippets from how i use it in django channels context

consumer.py

class FocusSessionConsumer(AsyncWebsocketConsumer):
    async def connect(self):
        self.user = self.scope["user"]
        self.username = self.scope["url_route"]["kwargs"]["username"]
        self.request = self._generate_request_metadata()
        await self.channel_layer.group_add(self.session_group_name, self.channel_name)  # type: ignore
        await self.accept()
        # othe code related to project
        logger.info(f"{self.user.username} connected to session '{self.session_id}'", extra={"request": self.request})
        await self.update_session_followers_list_to_all_clients()

    def _generate_request_metadata(self):
        request = HttpRequest()
        user_agent = str(self.scope["headers"][4][1].strip())
        try:
            ip_add = self.scope["client"][0]
        except Exception:
            ip_add = "-"
        try:
            user_os = user_agent.rsplit("(")[1].split(";")[0]
        except Exception:
            user_os = "-"
        request.META = {
            "REMOTE_ADDR": ip_add,
            "HTTP_SEC_CH_UA_PLATFORM": user_os,
        }
        request.user = self.user
        return request

Wrapping Up

With these custom filters, your logs will be bursting with useful info. Happy debugging, folks!

Remember, logging is like a good curry - it should be rich, flavorful, and help you find the source of the problem. Now go forth and log like a pro! 🚀📝

p.s - i am still learning so there could be some mistakes in this blog. please test your code after implementing.

This blog post will take you through the basics of Django's annotate function and then delve into some more advanced uses, helping you to fully leverage the power of Django's ORM in your projects.

The Basics

At its core, annotate() is about adding new, calculated fields to the objects in a queryset. Let's start with a simple example. Let's say we're working with two models - Blog and Comment:

from django.db import models

class Blog(models.Model):
    title = models.CharField(max_length=200)
    content = models.TextField()

class Comment(models.Model):
    blog = models.ForeignKey(Blog, on_delete=models.CASCADE, related_name='comments')
    content = models.TextField()
    created_at = models.DateTimeField(auto_now_add=True)

Now, if you want to find out how many comments each blog post has, you could use annotate() with the Count aggregation, like so:

from django.db.models import Count

# Get a queryset of all blogs
blogs = Blog.objects.all()

# Annotate each blog with the count of its related comments
blogs_with_counts = blogs.annotate(comment_count=Count('comments'))

for blog in blogs_with_counts:
    print(f"The blog '{blog.title}' has {blog.comment_count} comments.")

Here, Count is an aggregation function that counts the number of related objects. We're using it to count the number of related Comment objects and adding that count to each Blog object in the queryset under the field comment_count.

Beyond Basics: Dive into Advanced Usage

Django's annotate() function can do much more than just simple aggregations. You can use it to perform more complex queries and computations.

Conditional Expressions:

Django provides classes to represent SQL’s conditional expressions. Here’s an example using Case and When:

from django.db.models import Case, When, Count, IntegerField

blogs = Blog.objects.annotate(
    comment_count=Count('comments')
).annotate(
    has_many_comments=Case(
        When(comment_count__gt=10, then=1),
        default=0,
        output_field=IntegerField(),
    ),
)

This will annotate a QuerySet of blogs with a 'has_many_comments' field. The field is 1 if the blog has more than 10 comments, and 0 otherwise.

Expressions with F objects:

F expressions can be used in annotate to perform database operations without pulling the data into python.

from django.db.models.functions import Length

# This will annotate the QuerySet with the length of the title.
blogs = Blog.objects.annotate(title_length=Length('title'))

Using multiple aggregations:

You can use more than one aggregation function in your annotate clause.

from django.db.models import Count, Avg

# This will annotate the QuerySet with both the count of comments and the average id of comments.
blogs = Blog.objects.annotate(
    comment_count=Count('comments'),
    average_comment_id=Avg('comments__id')
)

Annotations on related models:

You can annotate related models to perform calculations that need to access related fields.

from django.db.models import Sum

# This will annotate each comment of each blog with the total views of the blog it belongs to.
comments = Comment.objects.select_related('blog').annotate(
    blog_views=Sum('blog__views')
)

Complex annotations with custom aggregates:

Django also allows you to define your own aggregate functions. This is more advanced and requires an understanding of how your database works.

These are just examples, but Django's ORM is incredibly powerful and flexible. You can create some very complex queries once you get a handle on how it all works.

In summary, Django's annotate is a powerful feature, allowing you to perform complex database queries and aggregations easily. It's a tool that can be a real game-changer once you learn how to use it effectively. Always remember to check Django's documentation, which is very comprehensive and includes many examples and explanations. Happy Coding!

1. Install the "PostgreSQL" extension in VS Code:

   • it should look like the below screenshot.👇

     

     Click "Install" to install the extension

2. Configure the remote PostgreSQL server to accept connections from your local machine. To do this, you will need to edit the pg_hba.conf file, which controls which clients are allowed to connect to the server. Follow these steps:

   • Connect to your Ubuntu VPS using SSH

   • Navigate to the pg_hba.conf file, which is usually located at /etc/postgresql/10/main/pg_hba.conf (the exact path may vary depending on your PostgreSQL version)

   • Edit the file using a text editor such as vi or nano:

    sudo vi pg_hba.conf

• under IPv4 local connections, set it to your ipaddress or 0.0.0.0/0 if your IP is dynamic.

  

1. Configure the remote PostgreSQL server to listen for connections on the network. By default, PostgreSQL only listens for connections on the local loopback interface (127.0.0.1). To allow connections from other hosts, you will need to edit the postgresql.conf file.

   • Navigate to the postgresql.conf file, which is usually located at /etc/postgresql/10/main/postgresql.conf (the exact path may vary depending on your PostgreSQL version)

   • Edit the file using a text editor such as vi or nano:

    sudo vi postgresql.conf

• Find the line that reads listen_addresses = 'localhost' and change it to listen_addresses = 'your_server_ipaddress' like below.

  

1. Restart the PostgreSQL server to apply the changes. You can do this by running the following command:

    sudo systemctl restart postgresql
   

2. Connect to the remote PostgreSQL server from VS Code. To do this, follow these steps:

   • Click on the Database icon in left of sidebar. it should like like below image.

     

   • Click on create new connection and you will see the screen like below image.

     

   • fill all the parameters including ones in ssh tunnel too.

   • click connect after filling everything.

   • now you will see all the database tables in sidebar.

     

Is this method secure 🤔

The method described above for connecting to a PostgreSQL database running on an Ubuntu VPS from a local machine using vs code is generally secure, as it uses the md5 authentication method and requires a valid username and password to access the database.

However, still there can be lot of things to make it secure & I don't have much info on how to make it most secure. but there are a few additional steps you can take to further secure the connection:

• Use a strong password for the PostgreSQL user. Make sure to use a password that is difficult to guess and not shared with any other accounts.

• Enable SSL encryption for the connection. To do this, you will need to edit the postgresql.conf file on the remote server and set the ssl parameter to on. You will also need to generate SSL certificates and configure them on both the client and server sides. Checkout this tutorial for more info on this step.

• Use a firewall to limit access to the PostgreSQL server. You can use a firewall such as iptables or ufw to allow connections only from trusted IP addresses or networks.

• Regularly update and patch the PostgreSQL server to keep it secure. Make sure to keep the server up to date with the latest security updates and patches.

you can do more research on making it more secure.

In conclusion, connecting to a PostgreSQL database from a local machine using a GUI tool such as pgAdmin or a VS Code extension like PostgreSQL by weijan can be a convenient way to manage and query the database. To do this, you will need to install the appropriate tool, configure the remote PostgreSQL server to accept connections, and establish a connection to the database using the hostname, port, username, and password. By following the steps outlined in this answer, you should be able to successfully connect to a PostgreSQL database from a local machine.

It is important to keep in mind that security is an important aspect of any database connection. To ensure that your connection is secure, it is recommended to use a strong password, enable SSL encryption, use a firewall to limit access to the server, and regularly update and patch the PostgreSQL server.

I hope this tutorial helps you. if you have any questions or suggestions, pls drop them in the comments.💬

If you want a Django developer who can bring your project to life, don't hesitate to contact me and let's discuss your needs! 😀

To use pre-commit tools, you will first need to install the pre-commit framework, which is a tool that allows you to manage and automate pre-commit hooks. You can install the pre-commit framework using pip:

pip install pre-commit

Once the pre-commit framework is installed, you can configure it to use specific pre-commit tools by creating a .pre-commit-config.yaml file in the root directory of your project. This file should specify the pre-commit hooks that you want to use, as well as any arguments or configuration options that you want to pass to these hooks.

For example, here is a .pre-commit-config.yaml file that configures the pre-commit framework to use the black and flake8 pre-commit tools:

exclude: "^docs/|/migrations/"
default_stages: [commit]

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml

  - repo: https://github.com/asottile/pyupgrade
    rev: v3.3.1
    hooks:
      - id: pyupgrade
        args: [--py310-plus]

  - repo: https://github.com/psf/black
    rev: 22.12.0
    hooks:
      - id: black

  - repo: https://github.com/PyCQA/isort
    rev: 5.11.4
    hooks:
      - id: isort

  - repo: https://github.com/PyCQA/flake8
    rev: 6.0.0
    hooks:
      - id: flake8
        args: ["--config=setup.cfg"]
        additional_dependencies: [flake8-isort]

In this example, the black tool is used to automatically format Python code using the Black code style, while the flake8 tool is used to check for syntax errors and style.

To use the pre-commit tools configured in your .pre-commit-config.yaml file, you will need to install the pre-commit framework's git hook. You can do this by running the following command from the root directory of your project:

pre-commit install

This will install a git hook that will run the configured pre-commit tools automatically before you commit any changes to your code.

You can also run the pre-commit tools manually at any time by using the pre-commit run command. This can be useful if you want to check your code for problems without actually committing it, or if you want to run the pre-commit tools on specific files or directories.

By using pre-commit tools, you can automate many of the tedious and error-prone tasks involved in code review and quality assurance, and can focus on the more important aspects of your project. Pre-commit tools can also help you enforce consistent coding standards and practices across your team, and can make it easier to maintain and update your code over time.

To know more about pre-commits, check this pre commit docs

If you want a Django developer who can bring your project to life, don't hesitate to contact me and let's discuss your needs! 😀

The .editorconfig file uses a simple INI-style format, with sections for each file pattern and properties for each setting. For example, a .editorconfig file might look something like this:

# Top-most EditorConfig file
root = true

# All files
[*]
indent_style = space
indent_size = 2
end_of_line = lf
charset = utf-8
trim_trailing_whitespace = true
insert_final_newline = true

# Python files
[*.py]
indent_size = 4

This .editorconfig file specifies that all files should use 2-space indentation, UTF-8 character encoding, and Unix-style line endings, and that trailing whitespace should be trimmed and a final newline should be inserted. It also specifies that Python files should use 4-space indentation.

To use the .editorconfig file with a particular editor or IDE, you will typically need to install a plugin or extension that adds support for EditorConfig. Once the plugin is installed, the editor or IDE will automatically read the .editorconfig file in the root directory of the project (or any of its parent directories) and apply the specified settings. This can help ensure that coding styles are consistent across different development environments and editors, even if different developers are using different editors.

EditorConfig is supported by a wide range of editors and IDEs, including Sublime Text, Visual Studio Code, and many others. By using a .editorconfig file, you can help ensure that your project's coding style is consistent and easy to maintain, regardless of the editor or IDE being used.

Summary 🎯

To summarize, the .editorconfig file is an essential tool for anyone looking to ensure consistent coding styles in their projects. With support from a wide range of editors and IDEs, .editorconfig is an easy and effective way to maintain a consistent coding style across different development environments. So if you want to improve the quality and consistency of your code, give .editorconfig a try today!

If you want a Django developer who can bring your project to life, don't hesitate to contact me and let's discuss your needs! 😀

Using super() can greatly simplify the process of working with inheritance in Django, allowing you to easily reuse code and customize your models, views, forms, and other components to fit your specific needs. In this article, we'll take a closer look at how to use the super() keyword in Django and explore some common use cases.

Inheriting attributes and methods in the __init__ method

One common use of super() in Django is in the __init__ method of a subclass. This method is called when an instance of the subclass is created, and it is used to initialize the attributes of the instance.

For example, consider a Django model called Product with a subclass called Book. The Product model might have fields for a title and a price, while the Book model might have an additional field for the author. In the __init__ method of the Book model, we can use super() to call the __init__ method of the Product model and pass in the title and price fields, while also adding the author field specific to the Book model:

class Product(models.Model):
    title = models.CharField(max_length=200)
    price = models.DecimalField(decimal_places=2, max_digits=10)

class Book(Product):
    author = models.CharField(max_length=200)

    def __init__(self, *args, **kwargs):
        # additional procesing before saving
        super().__init__(*args, **kwargs)

Using super().__init__(*args, **kwargs) in this way allows the Book model to inherit the title and price fields from the Product model, while also adding the author field specific to the Book model.

Overriding or extending methods in the subclass

Another common use of super() in Django is in the save method of a model. This method is called when the model instance is saved to the database, and it is used to persist any updates to the model instance.

For example, consider a Django model called Product with a save method that updates a last_modified field every time the model is saved. A subclass of Product called Book might want to override the save method to do some additional processing before saving the model. In this case, the save method of the Book model could use super() to call the save method of the Product model and pass in any additional arguments, while also doing the additional

class Product(models.Model):
    title = models.CharField(max_length=200)
    price = models.DecimalField(decimal_places=2, max_digits=10)
    last_modified = models.DateTimeField(auto_now=True)

    def save(self, *args, **kwargs):
        self.last_modified = timezone.now()
        self.save()

class Book(Product):
    author = models.CharField(max_length=200)

    def save(self, *args, **kwargs):
        # Do additional processing here
        super().save(*args, **kwargs)

In this revised version of the example code, the save method of the Product model saves the model instance after updating the last_modified field. When the save method of the Book model is called, it uses super().save(*args, **kwargs) to call the save method of the Product model, which saves the model instance with the updated last_modified field.

Using super() in a Django serializer

In Django, serializers are used to convert Django models or querysets into JSON format, allowing you to easily send data over the web or store it in a database. Serializers are typically used in combination with Django views and Django Rest Framework, a powerful toolkit for building APIs in Django.

You can use super() in a Django serializer to easily inherit attributes and methods from a parent serializer and customize or extend them as needed. For example, consider a Django serializer called ProductSerializer with a subclass called BookSerializer. The ProductSerializer might have fields for a title and a price, while the BookSerializer might have an additional field for the author. In the BookSerializer, we can use super() to call the ProductSerializer and pass in the title and price fields, while also adding the author field specific to the BookSerializer.

Using super() in Django template views

Django template views are used to render HTML templates and pass data to them for display. You can use super() in a Django template view to easily inherit attributes and methods from a parent template view and customize or extend them as needed.

One common use of super() in Django template views is in the get_context_data method, which is used to pass data to the template for rendering. For example, consider a Django template view called ProductView with a subclass called BookView. The ProductView might have a title variable, while the BookView might want to add an additional author variable. In the BookView, we can use super() to call the get_context_data method of the ProductView and pass in the title variable, while also adding the author variable specific to the BookView:

class ProductView(TemplateView):
    template_name = 'product.html'

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['title'] = 'Product'
        return context

class BookView(ProductView):
    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        context['author'] = 'John Doe'
        return context

Using super().get_context_data(**kwargs) in this way allows the BookView to inherit the title variable from the ProductView.

Using super() in Django forms

Django forms are used to validate and process user input. You can use super() in a Django form to easily inherit attributes and methods from a parent form and customize or extend them as needed.

One common use of super() in Django forms is in the validate method, which is used to perform validation on the form data. For example, consider a Django form called ProductForm with a subclass called BookForm. The ProductForm might have a field for a title, while the BookForm might want to add an additional author field and validate that the author field is not empty. In the BookForm, we can use super() to call the validate method of the ProductForm and pass in the cleaned_data dictionary, while also performing the additional validation for the BookForm:

class ProductForm(forms.Form):
    title = forms.CharField(max_length=200)

    def validate(self):
        cleaned_data = super().validate()
        title = cleaned_data.get('title')
        # Validate the title field
        if not title:
            raise forms.ValidationError('Please enter a title.')
        return cleaned_data

class BookForm(ProductForm):
    author = forms.CharField(max_length=200)

    def validate(self):
        cleaned_data = super().validate()
        author = cleaned_data.get('author')
        # Validate the author field
        if not author:
            raise forms.ValidationError('Please enter an author.')
        return cleaned_data

Using super().validate() in this way allows the BookForm to inherit the validation for the title field from the ProductForm, while also performing additional validation for the author field specific to the BookForm.

Conclusion

The super() keyword is a useful tool in Django for working with inheritance in classes. It allows a subclass to easily inherit attributes and methods from a parent class and customize or extend them as needed. Whether you're working with models, views, forms, or other components, super() can greatly simplify the process of working with inheritance in Django.

I hope this article has helped you understand how to use the super() keyword in Django and how it can make working with inheritance in your projects easier and more efficient. If you have any further questions or need additional clarification, don't hesitate to ask.

Disclaimer - I am not an expert & still learning. So, there may be some things which i may missed out or may be wrongly explained. As I got any comment about mistakes or figured out somewhere then i will correct it in blog.

1. Create a new user

• adduser yourusername

• usermod -aG sudo yourusername

• logout and login with your new user

2. Set the HostName

• hostnamectl set-hostname yourdomain.com

• hostname # check the hostname

• sudo nano /etc/hosts # add the hostname to the hosts file

  • host_ip_address yourdomain.com

3. Passwordless login

• on the local environment:

  • generate your ssh key if you don't have one

  • copy the public key

• on the remote environment:

  • create a .ssh folder

  • In that folder, create a file named authorized_keys like this .ssh/authorized_keys

  • paste the public key into the file

  • each key should be in a separate line. If you have multiple keys, you can paste them all into the file to allow multiple users to log in without a password

  • chmod 700 .ssh

  • chmod 600 .ssh/authorized_keys

  • sudo nano /etc/ssh/sshd_config # change PermitRootLogin & PasswordAuthentication to no

  • sudo service ssh restart

4. Install & configure UFW

sudo apt install ufw
sudo ufw allow ssh
sudo ufw enable
sudo ufw status

5. Install Dependencies

git clone yourproject
sudo apt update
sudo apt install python3.8 python3-pip python3.8-venv python3.8-dev build-essential libpq-dev python3-dev postgresql postgresql-contrib nginx curl

6. Configure Postgresql

sudo -u postgres psql
CREATE DATABASE my_db;
CREATE USER db_usr WITH ENCRYPTED PASSWORD 'some_secure_pass@pass';
ALTER ROLE db_usr SET client_encoding TO 'utf8';
ALTER ROLE db_usr SET default_transaction_isolation TO 'read committed';
GRANT ALL PRIVILEGES ON DATABASE my_db TO db_usr;

7. Configure Project

python3.8 -m venv env
pip install --upgrade pip
pip install -r requirements.txt
python manage.py migrate

8. Configure Gunicorn

sudo apt install gunicorn
sudo nano /etc/systemd/system/gunicorn.socket

Below is the content, you need to put in this file.

[Unit]
Description=gunicorn socket

[Socket]
ListenStream=/run/gunicorn.sock

[Install]
WantedBy=sockets.target

we also need a gunicorn service file. so enter to this command to create & edit that service file.

sudo nano /etc/systemd/system/gunicorn.service

add the below content to this file.

[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target

[Service]
User=saurav
Group=www-data
WorkingDirectory=/home/saurav/trends
EnvironmentFile=/home/saurav/trends/.base.env
ExecStart=/home/saurav/trends/env/bin/gunicorn \
        --access-logfile - \
        --workers 3 \
        --bind unix:/run/gunicorn.sock \
        trends.wsgi:application

[Install]
WantedBy=multi-user.target

Now Run below commands -

sudo systemctl start gunicorn.socket
sudo systemctl enable gunicorn.socket
sudo systemctl status gunicorn.socket
file /run/gunicorn.sock

Note: If you get an error like this: - debug it with sudo journalctl -u gunicorn.socket

8. Configure Nginix

sudo nano /etc/nginx/sites-available/trends

put below content in this file.

server {
    listen 8000;
    server_name 139.84.163.233;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        root /home/saurav/trends;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:/run/gunicorn.sock;
    }
}

sudo ln -s /etc/nginx/sites-available/trends /etc/nginx/sites-enabled

sudo nginx -t

sudo systemctl restart nginx

sudo ufw allow 'Nginx Full'

Congratulations 🎉🎊 if all steps are executed successfully, then your website should be live now.

Note: Some commands to reload/modify/debug services:

sudo systemctl restart gunicorn  # restart gunicorn
sudo systemctl daemon-reload  
sudo tail -F /var/log/nginx/error.log  # check nginix error logs
sudo systemctl status postgresql  # start your psql db server
sudo systemctl restart gunicorn.socket gunicorn.service  # restart gunicorn

If you are still having problem then check this great article by Digital Ocean.

https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-18-04

Disclaimer - I am not an expert & still learning. So, there may be some things which i may missed out or may be wrongly explained. As I got any comment about mistakes or figured out somewhere then i will correct it in blog.

🚧 Before Moving ahead, you must have a basic knowledge of Docker & Docker Compose.

Here i am going to use my already built django based app Movie Api which is using Postgres, Redis & ElasticSearch.

I will use a Docker compose file to keep every service in order as we can only starts our django server after postgres is ready and we can only starts elastic search after redis is ready. so these services are interdependent & I am going to link them properly so that they can start in an order, rather than just running independently and breaking things.

Before I start Explaining, have a look at my final Docker compose file.

version: '3.8'

services:
  django_server:
    build: .
    working_dir: /usr/src/app
    command: >
      sh -c "python manage.py wait_for_db &&
            python manage.py wait_for_elk &&
            python manage.py migrate &&
            apt install -y curl &&
            echo 'sending curl to Elastic hosts' &&
            curl -X GET ${ELK_CONNECTION_PROTOCOL}://${ELK_HOST}:${ELK_PORT} &&
            python manage.py generate_test_data 1000 &&
            gunicorn src.wsgi:application --bind 0.0.0.0:8000"
    ports:
      - 8000:8000
    env_file: .env
    depends_on:
      - elasticsearch7
      - postgresql_db
    container_name: django_server

  elasticsearch7:
    image: elasticsearch:7.17.5
    ports:
      - ${ELK_PORT}:${ELK_PORT}
      - 9300:9300
    container_name: ${ELK_HOST}
    restart: always
    environment:
      - discovery.type=single-node
      - node.name=django_flix
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

  postgresql_db:
    image: postgres
    ports:
      - 5432:5432
    restart: always
    env_file: .env
    container_name: ${POSTGRES_HOST}

In above Code, I have added depends_on to make services wait for another one before starting up. I have not very good understanding of docker, so i don't know why the django server always get started before even fully starting the dependent services. So, to overcame this issue i made a custom command in django which stops the server until the Database and ElasticSearch are fully started.

here is code for both of the files where i write custom commands.

"""
Django Command to wait for DB to be available
"""
import time

from django.core.management import BaseCommand
from django.db.utils import OperationalError as DjangoDbUtilsOperationalError
from psycopg2 import OperationalError as Psycopg2OperationalError


class Command(BaseCommand):
    """Django command to pause execution until database is available"""

    def handle(self, *args, **options):
        self.stdout.write("Waiting for database...")
        db_is_up = False
        while not db_is_up:
            try:
                # try to connect with db
                self.check(databases=["default"])
                db_is_up = True
                self.stdout.write(self.style.SUCCESS("Database is available"))
            except (Psycopg2OperationalError, DjangoDbUtilsOperationalError):
                self.stdout.write(
                    self.style.WARNING(
                        "Database is still not ready. will try in 1 second"
                    )
                )
                time.sleep(1)

And this one below is for waiting for elastic search to start.

"""
Django Command to wait for DB to be available
"""
import time

from django.conf import settings
from django.core.management import BaseCommand
from elasticsearch import Elasticsearch


class Command(BaseCommand):
    """Django command to pause execution until database is available"""

    def handle(self, *args, **options):
        self.stdout.write("Waiting for ElasticSearch...")
        elk_is_up = False
        host = settings.ELK_HOST
        port = settings.ELK_PORT
        while not elk_is_up:
            try:
                # try to connect with db
                self.stdout.write(
                    self.style.WARNING(
                        "trying to connect with elk at {}:{}".format(host, port)
                    )
                )
                es = Elasticsearch([{"host": host, "port": port}])
                es.cluster.health(wait_for_status="yellow", request_timeout=1)
                self.stdout.write(self.style.SUCCESS("✅ ElasticSearch is available"))
                elk_is_up = True
            except Exception:
                self.stdout.write(
                    self.style.WARNING(
                        "❌ ElasticSearch is still not ready. will try in 5 second"
                    )
                )
                time.sleep(5)

I also uses a enviroment file so that i can pass these values to docker compose file instead of defining them in the key value pair in docker compose file.

export DEBUG=1
export POSTGRES_DB="postgres"
export POSTGRES_USER="postgres"
export POSTGRES_PASSWORD="postgres"
export POSTGRES_HOST="postgresql_db_latest"
export POSTGRES_PORT=5432
export ES_JAVA_OPTS="-Xms8g -Xmx8g"
export ELK_HOST="elasticsearch_v7"
export ELK_PORT="9200"
export ELK_INDEX_NAME="django_flix"
export ELK_CONNECTION_PROTOCOL="http"

This way I was able to link these services and starts them in a order.

Helpful Links

• Django
• ElasticSearch 7 Installation guide
• Django Custom Commands
• Docker Compose
• Link to Code on GitHub

Disclaimer - I am not an expert & still learning. So, there may be some things which i may missed out or may be wrongly explained. As I got any comment about mistakes or figured out somewhere then i will correct it in blog.

In this Blog, I am going to show you how can return only those fields from the API which user have requested by passing them in URL as parameters. this way your API users can only request those fields which they need & this will remove some unnecessary clutter from your API response.

if you don't understand about URL parameters then head toward this link for some reference.

if you are a visual learner then checkout the video version of this blog.

For Reference, I will be using a demo API which I created in the previous Blog. you can find the code for that API here.

Now, first pass some params in the URL Bar of your browser & then hit enter.

As you can see, even after passing some fields in URL Bar, the API is still returning all fields. Which is obvious as we haven't implemented any logic for that yet.

Now, let's implement that logic. For that, we will be using our MovieHaystackSerializer from previous Blog Code. Now you will see some different code than what you usually write in your Serializer. Just Ignore new things as of now & only focus on __init__ and fields of our Serializer. So, first, we will edit our MovieHayStackSerializer by overiding the __init__ method.

# 3rd party imports
from drf_haystack.serializers import HaystackSerializer

# local imports
from .search_indexes import MovieIndex


class MovieHayStackSerializer(HaystackSerializer):
    class Meta:
        # The `index_classes` attribute is a list of which search indexes
        # we want to include in the search.
        index_classes = [MovieIndex]

        # The `fields` contains all the fields we want to include.
        # NOTE: Make sure you don't confuse these with model attributes. These
        # fields belong to the search index!
        fields = [
            "title",
            "description",
            "year",
            "rating",
            "global_ranking",
            "length",
            "revenue",
            "genre",
            "country",
            "director",
        ]

    def __init__(self, *args, **kwargs):
        super(MovieHayStackSerializer, self).__init__(*args, **kwargs)
        # get the query params
        fields_in_url_params = self.context["request"].GET.get("fields", None)
        if fields_in_url_params:
            # split the params from comma to make a list
            fields_in_url_parmas_list = fields_in_url_params.split(",")
            existing_fields = set(self.fields)  # -> keys of all fields

            # get all the fields which are not in params but declared in meta class above
            # our_fields = {'title', 'description', 'year', '....'}
            # fields_from_urls = {'title', 'year'}
            # operation = our_fields - fields_from_url => {'description', '.....'}
            fields_to_remove = existing_fields - set(fields_in_url_parmas_list)

            # now since we have fields 
            for field in fields_to_remove:
                self.fields.pop(field)  # remove this field

Now, let's see what we have done here. First, we have overrided the __init__ method of our Serializer. In this method, we are getting the query params from the URL & then we are splitting them by comma to make a list. After that, we are getting all the fields which are declared in our Meta class. Now, we are getting the difference between these two lists. The difference will be the fields which we have to remove from our Serializer. After that, we are removing those fields from our Serializer.

Now, let's see what happens when we pass some fields in URL Bar.

As you can see, now we are only getting those fields which we have passed in URL Bar. This is how you can return only those fields from the API which user have requested by passing them in URL as parameters. Link for Repo - https://github.com/selftaughtdev-me/movie-search-api

If you have any questions or suggestions, feel free to comment below. I will be happy to help you. Also, if you like this Blog, then please share it with your friends & colleagues. you can also follow me on Twitter & LinkedIn for getting updates about my new Blogs.