Returning BigQuery Results in Apache Arrow Format

[Apache Arrow] is a high performance, in-memory binary data format. It is quickly emerging as a de facto standard, and BigQuery has added native support for it. Utilizing this capability allows you to avoid manual JSON encoding and decoding on both the server and the client.

Compare the following code to the version in [Using Google BigQuery from Django], from which it was adapted, and note the reduction in complexity.

from django.http import HttpResponse
from google.cloud import bigquery

def index(request):
    query = """
      SELECT * FROM
      `bigquery-public-data.new_york_taxi_trips.tlc_yellow_trips_2018`
      LIMIT 1000
    """

    bq_client = bigquery.Client()
    query_job = bq_client.query(query)
    arrow_table = query_job.to_arrow()

    sink = pa.BufferOutputStream()
    writer = pa.ipc.new_stream(sink, arrow_table.schema)
    for batch in arrow_table.to_batches():
      writer.write_batch(batch)
    writer.close()
    buf = sink.getvalue()
    return HttpResponse(
        buf.to_pybytes(),
        content_type="application/octet-stream")