High Performance Web Graphics

You're effectively building the most performant pipeline you can from the database to a user’s GPU.

[Creating a Google Cloud Platform Project for High Performance Web Graphics]

• Database/Storage
  • Anything from BiqQuery to Postgres/MySQL to NoSQL
  • Hybrid storage mixes denormalized and serialized data to accommodate both querying and high performance
    • Store queryable fields as columns or properties
    • Store full serialized row/document for fast retrieval and sending
      • Serialized [Protocol Buffers]
      • [Apache Arrow] Columnar Format
  • [Create a New Firebase Application]
  • Download and convert sample data
    • [Free Datasets for High Performance Web Graphics Development]
• Server
  • [Python] is a common choice because of its prevalence in data science
  • [pyarrow] are the Python bindings for [Apache Arrow]
    • TBD: Optimal process for getting to Arrow structures in Python
    • Arrow Buffers have a to_pybytes() function that returns a Python buffer
  • Serialize tables/record batches as buffers, attach to proto, and ship them to the client
  • [How to Serve Binary Protocol Buffer Data from Django]
  • [How to Construct Protocol Buffer Instances in Python]
  • Streaming?
    • gRPC?
    • WebSockets?
• Client
  • On the client it's one line to deserialize the protobuf and one line to deserialize each record batch
  • Both operations should be zero copy and lazy
  • [How to Load Binary Protocol Buffer Data in Angular]
  • [[Using Apache Arrow with Angular]]
  • [Typed Arrays] pop up here and there
  • GPU-based visualization with tools like [[deck.gl]]
  • GLSL [shaders] for custom graphics
• Misc.
  • [Protocol Buffers] as data definition and transport with bytes field(s) for serialized data
  • [Rendering Millions of Interactive Points on the web]
  • Geospatial data can be encoded in a number of [geographic data formats]
  • 3 Data Locations:
    • At Rest
    • On the Wire
    • Client Memory