Senior or Staff ML Systems Engineer, LLMs
TRM Labs, Washington, District of Columbia, United States
Build a Safer World.
TRM Labs provides blockchain analytics and AI solutions to help law enforcement and national security agencies, financial institutions, and cryptocurrency businesses detect, investigate, and disrupt crypto‑related fraud and financial crime. TRM’s blockchain intelligence and AI platforms can trace the source and destination of funds, identify illicit activity, build cases, and construct an operating picture of threats. TRM is trusted by leading agencies and businesses worldwide who rely on TRM to enable a safer, more secure world for all.
Role Overview
As a Senior or Staff ML Systems Engineer – LLM, you will core responsibilities include building and scaling the technical infrastructure for AI/ML systems. You will set the pace for AI deployments, ensuring they are fast, safe, and scalable.
Responsibilities
Build reusable CI/CD workflows for model training, evaluation, and deployment – integrating Langfuse, GitHub Actions, experiment tracking, and more.
Automate model versioning, approval workflows, and compliance checks across environments.
Create a modular and scalable AI infrastructure stack – including vector databases, feature stores, model registries, and observability tooling.
Partner with engineering and data science to embed AI models and agents into real‑time applications and workflows.
Continuously evaluate and integrate state‑of‑the‑art AI tools (e.g. LangChain, LlamaIndex, vLLM, MLflow, BentoML).
Drive AI reliability and governance, enabling experimentation while ensuring compliance, security, and uptime.
Build and enhance AI/ML model performance.
Ensure data accuracy, consistency, and reliability for improved model training and inference.
Deploy infrastructure to support offline and online evaluation of LLMs and agents – including regression testing, cost monitoring, and human‑in‑the‑loop workflows.
Enable researchers to iterate quickly by providing sandboxes, dashboards, and reproducible environments.
What We’re Looking For
Write high‑quality, maintainable software – primarily in Python, but engineering ability is valued over language familiarity.
Strong background in scalable infrastructure, including:
Containerization and orchestration (Docker, Kubernetes)
Infrastructure‑as‑code and deployment (Terraform, CI/CD pipelines)
Monitoring and logging frameworks (Datadog, Prometheus, OpenTelemetry)
Understand and implement ML Ops best practices, including:
Model versioning and rollback strategies
Automated evaluation and drift detection
Scalable model and agent serving infrastructure (vLLM, Triton, BentoML)
Deploy and maintain LLM and agentic workflows in production, including:
Monitoring cost, latency, and performance
Capturing traces for analysis and debugging
Optimizing prompt/response flows with real‑time data access
Demonstrate strong ownership and pragmatism, balancing infrastructure elegance with iterative delivery and measurable impact.
Compensation
Estimated base salary range: $200,000 – $275,000 US base.
Role may be eligible for TRM’s equity plan.
Legal Statement
TRM Labs is an equal opportunity employer. We welcome applicants from all backgrounds and do not discriminate on the basis of race, color, religion, sex, gender identity, sexual orientation, national origin, age, disability, or any other protected characteristic.
#J-18808-Ljbffr
TRM Labs provides blockchain analytics and AI solutions to help law enforcement and national security agencies, financial institutions, and cryptocurrency businesses detect, investigate, and disrupt crypto‑related fraud and financial crime. TRM’s blockchain intelligence and AI platforms can trace the source and destination of funds, identify illicit activity, build cases, and construct an operating picture of threats. TRM is trusted by leading agencies and businesses worldwide who rely on TRM to enable a safer, more secure world for all.
Role Overview
As a Senior or Staff ML Systems Engineer – LLM, you will core responsibilities include building and scaling the technical infrastructure for AI/ML systems. You will set the pace for AI deployments, ensuring they are fast, safe, and scalable.
Responsibilities
Build reusable CI/CD workflows for model training, evaluation, and deployment – integrating Langfuse, GitHub Actions, experiment tracking, and more.
Automate model versioning, approval workflows, and compliance checks across environments.
Create a modular and scalable AI infrastructure stack – including vector databases, feature stores, model registries, and observability tooling.
Partner with engineering and data science to embed AI models and agents into real‑time applications and workflows.
Continuously evaluate and integrate state‑of‑the‑art AI tools (e.g. LangChain, LlamaIndex, vLLM, MLflow, BentoML).
Drive AI reliability and governance, enabling experimentation while ensuring compliance, security, and uptime.
Build and enhance AI/ML model performance.
Ensure data accuracy, consistency, and reliability for improved model training and inference.
Deploy infrastructure to support offline and online evaluation of LLMs and agents – including regression testing, cost monitoring, and human‑in‑the‑loop workflows.
Enable researchers to iterate quickly by providing sandboxes, dashboards, and reproducible environments.
What We’re Looking For
Write high‑quality, maintainable software – primarily in Python, but engineering ability is valued over language familiarity.
Strong background in scalable infrastructure, including:
Containerization and orchestration (Docker, Kubernetes)
Infrastructure‑as‑code and deployment (Terraform, CI/CD pipelines)
Monitoring and logging frameworks (Datadog, Prometheus, OpenTelemetry)
Understand and implement ML Ops best practices, including:
Model versioning and rollback strategies
Automated evaluation and drift detection
Scalable model and agent serving infrastructure (vLLM, Triton, BentoML)
Deploy and maintain LLM and agentic workflows in production, including:
Monitoring cost, latency, and performance
Capturing traces for analysis and debugging
Optimizing prompt/response flows with real‑time data access
Demonstrate strong ownership and pragmatism, balancing infrastructure elegance with iterative delivery and measurable impact.
Compensation
Estimated base salary range: $200,000 – $275,000 US base.
Role may be eligible for TRM’s equity plan.
Legal Statement
TRM Labs is an equal opportunity employer. We welcome applicants from all backgrounds and do not discriminate on the basis of race, color, religion, sex, gender identity, sexual orientation, national origin, age, disability, or any other protected characteristic.
#J-18808-Ljbffr