spiderweb/docs

spiderweb

As a professional web developer focusing on arcane uses of Django for arcane purposes, it occurred to me a little while ago that I didn't actually know how a web framework worked.

So I built one.

This is spiderweb, a WSGI-compatible web framework that's just big enough to hold a spider. When building it, my goals were simple:

• Learn a lot
• Create an unholy blend of Django and Flask
• Not look at any existing code¹. Go off of vibes alone and try to solve all the problems I could think of in my own way

Warning

This is a learning project. It should not be used for production without heavy auditing. It's not secure. It's not fast. It's not well-tested. It's not well-documented. It's not well-anything. It's a learning project.

That being said, it's fun and it works, so I'm counting that as a win.

[!TIP|style:flat] To jump in with both feet, head over to the quickstart!

Design & Usage Decisions

There are a couple of things that I feel strongly about that have been implemented into Spiderweb.

Deliberate Responses

In smaller frameworks, it's often the case that "how to return the response" is guessed based on the data that you return from a view. Take this view for example:

def index(request):
    return "Hi"

In this function, we return a string, and as such there are some assumptions that we make about what the response looks like:

• a content-type header of "text/plain"
• that we are wanting to return raw HTML
• no post-processing of the response is needed

While assumptions are nice for getting running quickly, I think that there is a balance between assuming what the user wants and making them define everything. See how Spiderweb handles this:

from spiderweb.response import HttpResponse

def index(request):
    return HttpResponse("Hi")

In this case, we improve readability of exactly what type of response we expect (raw HTML, a template response, a JSON-based response, etc.) and we give the developer the tools that they need to modify the response beforehand.

The response object has everything that it needs immediately available: headers, cookies, status codes, and more. All can be modified before sending, but providing this data for the opportunity to change is what's important.

Spiderweb provides five types of responses out of the box:

• HttpResponse
• FileResponse
• RedirectResponse
• TemplateResponse
• JSONResponse

Read more about responses in Spiderweb

Database Agnosticism (Mostly)

One of the largest selling points of Django is the Django Object Relational Mapper (ORM); while there's nothing that compares to it in functionality, there are many other ORMs and database management solutions for developers to choose from.

In order to use a database internally (and since this is not about writing an ORM too), Spiderweb depends on peewee, a small ORM. Applications using Spiderweb are more than welcome to use peewee models with first-class support or use whatever they're familiar with. Peewee supports PostgreSQL, MySQL, Sqlite, and CockroachDB; if you use one of these, Spiderweb can create the tables it needs in your database and stay out of the way. By default, Spiderweb creates a sqlite database in the application directory for its own use.

Read more about the using a database in Spiderweb

Easy to configure

Configuration is handled in different ways across different frameworks; in Flask, a simple dictionary is used that you can modify:

# https://flask.palletsprojects.com/en/3.0.x/config/
app = Flask(__name__)
app.config['TESTING'] = True

This works, but suffers from the problem of your IDE not being able to see all the possible options that are available. Django solves this by having a settings.py file:

# https://docs.djangoproject.com/en/5.1/topics/settings/
ALLOWED_HOSTS = ["www.example.com"]
DEBUG = False
DEFAULT_FROM_EMAIL = "webmaster@example.com"

Simply having these declared in a place that Django can find them is enough, and they Just Work™️. This also works, but can be very verbose.

Spiderweb takes a middle ground approach: it allows you to declare framework-first arguments on the SpiderwebRouter object, and if you need to pass along other data to other parts of the system (like custom middleware), you can do so by passing in any keyword argument you'd like to the constructor.

from spiderweb import SpiderwebRouter
from peewee import SqliteDatabase

app = SpiderwebRouter(
  db=SqliteDatabase("myapp.db"),
  port=4500,
  session_cookie_name="myappsession",
  my_middleware_data="Test!"
)

In this example, db, port, and session_cookie_name are all arguments that affect the server, but my_middleware_data is something new. It will be available in app.extra_data and is also available through middleware and on the request object later.

tl;dr: what can Spiderweb do?

Here's a non-exhaustive list of things this can do:

• Function-based views
• Optional Flask-style URL routing
• Optional Django-style URL routing
• URLs with variables in them a lá Django
• Full middleware implementation
• Limit routes by HTTP verbs
• Custom error routes
• Built-in dev server
• Gunicorn support
• HTML templates with Jinja2
• Static files support
• Cookies (reading and setting)
• Optional append_slash (with automatic redirects!)
• CSRF middleware
• CORS middleware
• Optional POST data validation middleware with Pydantic
• Session middleware with built-in session store
• Database support (using Peewee, but you can use whatever you want as long as there's a Peewee driver for it)
• Tests (currently a little over 80% coverage)

---

1. I mostly succeeded. The way that I'm approaching this is that I did my level best, then looked at (and copied) existing solutions where necessary. At the time of this writing, I did all of it solo except for the CORS middleware. Read more about it here. ↩︎