Django, Python, Linux, Dogs

Stream a response from an external server without exposing endpoints

Not a hack (but sloppy and fast)

February 1st, 2017

In the world of API services it still feels like sloppiness and loose ends are par for the course. Take Amazon Web Services' API Gateway, intended to allow for quick building of scalable APIs connected to the AWS Relational Database Service. That all sounds well and good, but AWS authentication is set to rely on CORS (Cross-Origin Resource Sharing) and/or IAM roles (which are internal to AWS and don't extend to clients as it is). The issue arises, though, that selective exposure to the API can be difficult without roping in more services. 

So, let's say you want controlled access to ANY external service and that you don't want to expose the service itself. There's not much you can do other than protect by domain. Plus, if we're talking about CORS, this is a big flowchart. That's never good. And then you'll end up asking living human beings questions like "I can't seem to work out CORS issues on my bucket" and they'll either think you're an idiot or a loon.  

A few months ago I was building the demo application for a startup with massive amounts of data and an API Gateway implementation that solely used CORS as protection. Initially this was seen as a difficulty that we wouldn't be able to work through for months. Showing off the demo itself required installation of a browser extension. It was less than ideal and not something you wanted to explain in a fundraising meeting. However, once we could call the API w/ our 'piggyback system', we had a lot of luck with investors. 

So, here's a recipe I used to connect to an API Gateway instance that sat protected by a secret key. With the use of Django's authentication and regex, it's easy enough to restrict access to that tasty JSON in a much more granular and sane way than you would otherwise do.

from django.conf import settings
from django.http import StreamingHttpResponse
from django.shortcuts import Http404

import requests

# In my case, the domain/root url and key used for protection were stored in
API_URL = getattr(settings, 'API_URL')   # In my case, a GET param appended to the API Gateway url 
API_KEY = getattr(settings, 'API_KEY')    # Url/IP for your API Gateway

def api_gateway_portal(request, path=''):
    | Makes a request to an external API (via API Gateway) while supplying the
        necessary access key. The request is streamed back immediately upon
    :param request:
    :param path: The path of the API url, minus domain. For example, to reach
        `{API url}/all/MyEndpoint`, path would equal 'all/MyEndpoint'
    :return: Streaming JSON http response from the API
    :rtype: StreamingHttpResponse

    # My, what a lovely place to check for permissions, etc. 
    if not any([
        raise Http404
    if not path.endswith('/'):
        path += "/"

    r = requests.get(API_URL % (path,))

    # Stream the response from the API Gateway as received; Iterate and send in chunks!
    response = StreamingHttpResponse(
        (chunk for chunk in r.iter_content(512 * 1024)),
        content_type='application/json'    # Enter the suitable content type you are using

    return response

from django.conf.urls import url
from . import views

urlpatterns = [
    url(r'^(?P<path>.*)$', views.api_gateway_portal, name="portal"),  

This isn't great for production or high-traffic situations, but it's quick, easy, reliable, and far faster than I expected (generally adding 25-30% additional load time for API retrieval). 

Tags: API, AWS, Django


Want to work together? Send an email.


In addition to building web and in-house applications and systems for Thermaline, I'm on Twitter, LinkedIn and Instagram. Let's talk!

©2018 Ian Price / Red Mountain GIS