You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
|
2 weeks ago | |
---|---|---|
modules/lambda | 2 weeks ago | |
src | 2 weeks ago | |
.gitignore | 2 weeks ago | |
.terraform.lock.hcl | 2 weeks ago | |
927-challenge-architecture.png | 2 weeks ago | |
README.md | 2 weeks ago | |
api_gw.frontend.tf | 2 weeks ago | |
api_gw.load.tf | 2 weeks ago | |
api_gw.metrics.tf | 2 weeks ago | |
api_gw.tf | 2 weeks ago | |
athena.tf | 2 weeks ago | |
frontend.tf | 2 weeks ago | |
load.tf | 2 weeks ago | |
main.tf | 2 weeks ago | |
metrics.tf | 2 weeks ago | |
providers.tf | 2 weeks ago | |
s3.tf | 2 weeks ago | |
terraform_graph.png | 2 weeks ago |
README.md
Technical Challenge Repo link
https://code.ndumas.com/ndumas/senior-platform-enginner-927-technical-challenge/
Architecture diagrams are included in my repository above.
Justifications
- Athena
- Less up-front configuration involved.
- AWS wants to lock you into their ecosystem so it'll be cheaper.
- In the future, we can switch from CSV to JSON or Parquet or mix and match without any overhead like rewriting the /load endpoints to use a different SQL flavor or parsing library.
- API Gateway
- Rate limiting and caching built-in
- Abstracts interfacing with the API away from raw lambda invocation URLs
- Can be mapped to a domain
Challenge Deliverables
- Architecture diagram
- saved as a PNG
- Explain decisions and trade-offs
- Document in README.md
- Terraform repo scaffolding out the service
Air Traffic Metrics
- Ingest raw data into S3
- S3 Bucket
- Bucket best practices
- Buckets aren't public
- Only
load
lambda can add objects - Only Athena can read objects
- ????
- Bucket best practices
- S3 Bucket
- Database of my choice
- Athena?
- Less up-front configuration involved. I hope.
- AWS wants to lock you into their ecosystem so it'll be cheaper.
- In the future, we can switch from CSV to JSON or Parquet or mix and match without any overhead.
- Aurora (Postgresql)
- Aurora (Mysql)
- Athena?
- API to interact with the data
- API Gateway
- Policy that grants it lambda invocation rights
/load
endpoint- Python lambda to upload data to bucket
- Example data: https://zenodo.org/records/5377831
- Python lambda to upload data to bucket
/metrics
endpoint- Python lambda that returns the following:
- row_count: number of rows in table for full data set
- last_transponder_seen_at: maximum value for
lastseen
- most_popular_destination: most seen value for destination
- count_of_unique_transponders: unique count of the icao24 field
- Execution role that grants it access to athena database
- Python lambda that returns the following:
- API Gateway
- Frontend to display the data
- Python lambda that renders an HTML page that displays above metrics