ndumas
/
senior-platform-enginner-927-technical-challenge
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
16 Commits
1 Branch
0 Tags
384 KiB
HCL 99.8%
Python 0.2%
 
 
main
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '113d71e42a'
${ noResults }
Go to file
Nick Dumas 113d71e42a We should be POSTing files to the load endpoint 10 months ago
modules/lambda Add needed outputs to lambda module 10 months ago
src Modularize the lambdas 10 months ago
.gitignore Ignore generated files 10 months ago
.terraform.lock.hcl Pin archive provider version 10 months ago
README.md Justify Athena usage some more 10 months ago
api_gw.load.tf We should be POSTing files to the load endpoint 10 months ago
api_gw.metrics.tf API Gateway 10 months ago
api_gw.tf API Gateway 10 months ago
athena.tf Sketching out more infra 10 months ago
frontend.tf Modularize the lambdas 10 months ago
load.tf Modularize the lambdas 10 months ago
main.tf API Gateway 10 months ago
metrics.tf Modularize the lambdas 10 months ago
providers.tf Sketching out more infra 10 months ago
s3.tf Modularize the lambdas 10 months ago

README.md

Challenge Deliverables

  • Architecture diagram
    • saved as a PNG
  • Explain decisions and trade-offs
    • Document in README.md
  • Terraform repo scaffolding out the service

Air Traffic Metrics

  • Ingest raw data into S3
    • S3 Bucket
      • Bucket best practices
        • Buckets aren't public
        • Only load lambda can add objects
        • Only Athena can read objects
        • ????
  • Database of my choice
    • Athena?
      • Less up-front configuration involved. I hope.
      • AWS wants to lock you into their ecosystem so it'll be cheaper.
      • In the future, we can switch from CSV to JSON or Parquet or mix and match without any overhead.
    • Aurora (Postgresql)
    • Aurora (Mysql)
  • API to interact with the data
    • API Gateway
      • Policy that grants it lambda invocation rights
    • /load endpoint
    • /metrics endpoint
      • Python lambda that returns the following:
        • row_count: number of rows in table for full data set
        • last_transponder_seen_at: maximum value for lastseen
        • most_popular_destination: most seen value for destination
        • count_of_unique_transponders: unique count of the icao24 field
      • Execution role that grants it access to athena database
  • Frontend to display the data
    • Python lambda that renders an HTML page that displays above metrics