ndumas
/
senior-platform-enginner-927-technical-challenge
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
24 Commits
1 Branch
0 Tags
384 KiB
HCL 99.8%
Python 0.2%
 
 
main
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
Go to file
Nick Dumas 1b7b66e272 And this too 2 weeks ago
modules/lambda Add needed outputs to lambda module 2 weeks ago
src Modularize the lambdas 2 weeks ago
.gitignore Ignore generated files 2 weeks ago
.terraform.lock.hcl Pin archive provider version 2 weeks ago
927-challenge-architecture.png Add architecture diagram 2 weeks ago
README.md And this too 2 weeks ago
api_gw.frontend.tf Reference frontend in the API gateway 2 weeks ago
api_gw.load.tf We should be POSTing files to the load endpoint 2 weeks ago
api_gw.metrics.tf API Gateway 2 weeks ago
api_gw.tf API Gateway 2 weeks ago
athena.tf Sketching out more infra 2 weeks ago
frontend.tf Modularize the lambdas 2 weeks ago
load.tf Modularize the lambdas 2 weeks ago
main.tf API Gateway 2 weeks ago
metrics.tf Modularize the lambdas 2 weeks ago
providers.tf Sketching out more infra 2 weeks ago
s3.tf Modularize the lambdas 2 weeks ago
terraform_graph.png Reference frontend in the API gateway 2 weeks ago

README.md

Technical Challenge Repo link

https://code.ndumas.com/ndumas/senior-platform-enginner-927-technical-challenge/

Architecture diagrams are included in my repository above.

Justifications

  • Athena
    • Less up-front configuration involved.
    • AWS wants to lock you into their ecosystem so it'll be cheaper.
    • In the future, we can switch from CSV to JSON or Parquet or mix and match without any overhead like rewriting the /load endpoints to use a different SQL flavor or parsing library.
  • API Gateway
    • Rate limiting and caching built-in
    • Abstracts interfacing with the API away from raw lambda invocation URLs
    • Can be mapped to a domain

Challenge Deliverables

  • Architecture diagram
    • saved as a PNG
  • Explain decisions and trade-offs
    • Document in README.md
  • Terraform repo scaffolding out the service

Air Traffic Metrics

  • Ingest raw data into S3
    • S3 Bucket
      • Bucket best practices
        • Buckets aren't public
        • Only load lambda can add objects
        • Only Athena can read objects
        • ????
  • Database of my choice
    • Athena?
      • Less up-front configuration involved. I hope.
      • AWS wants to lock you into their ecosystem so it'll be cheaper.
      • In the future, we can switch from CSV to JSON or Parquet or mix and match without any overhead.
    • Aurora (Postgresql)
    • Aurora (Mysql)
  • API to interact with the data
    • API Gateway
      • Policy that grants it lambda invocation rights
    • /load endpoint
    • /metrics endpoint
      • Python lambda that returns the following:
        • row_count: number of rows in table for full data set
        • last_transponder_seen_at: maximum value for lastseen
        • most_popular_destination: most seen value for destination
        • count_of_unique_transponders: unique count of the icao24 field
      • Execution role that grants it access to athena database
  • Frontend to display the data
    • Python lambda that renders an HTML page that displays above metrics