Lead Observability Engineer (New Relic)

Description

We are looking for a Lead Observability Engineer to build an Enterprise Agent Development Platform — a production-grade, cloud-native ecosystem that enables engineering teams to define, orchestrate, deploy, and observe AI agents at scale. The platform standardizes agent development across the organization using LangGraph and Strands Agents on AWS AgentCore Runtime. The initiative spans agent framework design, runtime architecture, marketplace integration, CI/CD automation, and enterprise-grade observability — reducing agent development from months to days while enforcing consistent security, quality, and governance standards. Responsibilities Own the multi-tenant New Relic consolidation layer aggregating telemetry across the agent estate Administer New Relic at scale, including management account setup and cross-account configuration Build and maintain cross-account NRQL queries that aggregate telemetry from multiple project tenants Develop dashboards-as-code and alert-policy-as-code using the Terraform New Relic provider and Crossplane Configure New Relic alert policies routed to Opsgenie, with future migration to ServiceNow Consume OpenTelemetry / ADOT data through OTLP and Firehose ingestion into New Relic Design metric dimensions for large agent estates, ensuring stable, low-cardinality facets Maintain segregation across Technical, Business, and Audit observability models Requirements 6+ years of experience in observability or SRE engineering Hands-on experience consolidating observability/telemetry for AI agent estates in a production agentic AI project, such as multi-tenant New Relic (or equivalent) aggregation of agent telemetry, cost/token attribution for GenAI workloads, or AgentCore GenAI Observability integration Expertise in New Relic (or equivalent SaaS observability platforms) operated as a multi-tenant consolidation layer Skills in cross-account / cross-tenant telemetry aggregation and reporting Familiarity with working within a Technical / Business / Audit observability segregation model Proficiency in NRQL, dashboards-as-code, and alert-policy-as-code practices Knowledge of OpenTelemetry / ADOT consumption fundamentals Nice to have Familiarity with AWS Bedrock AgentCore observability (Transaction Search, GenAI Observability) Background in Kinesis Firehose metric / log forwarding pipelines (CloudWatch → New Relic) Knowledge of cost / token attribution modeling for GenAI workloads Skills in Crossplane-based provisioning of observability resources

Requirements

• 6+ years of experience in observability or SRE engineering

• Hands-on experience consolidating observability/telemetry for AI agent estates in a production agentic AI project, such as multi-tenant New Relic (or equivalent) aggregation of agent telemetry, cost/token attribution for GenAI workloads, or AgentCore GenAI Observability integration

• Expertise in New Relic (or equivalent SaaS observability platforms) operated as a multi-tenant consolidation layer

• Skills in cross-account / cross-tenant telemetry aggregation and reporting

• Familiarity with working within a Technical / Business / Audit observability segregation model

• Proficiency in NRQL, dashboards-as-code, and alert-policy-as-code practices

• Knowledge of OpenTelemetry / ADOT consumption fundamentals

Responsibilities

• Own the multi-tenant New Relic consolidation layer aggregating telemetry across the agent estate

• Administer New Relic at scale, including management account setup and cross-account configuration

• Build and maintain cross-account NRQL queries that aggregate telemetry from multiple project tenants

• Develop dashboards-as-code and alert-policy-as-code using the Terraform New Relic provider and Crossplane

• Configure New Relic alert policies routed to Opsgenie, with future migration to ServiceNow

• Consume OpenTelemetry / ADOT data through OTLP and Firehose ingestion into New Relic

• Design metric dimensions for large agent estates, ensuring stable, low-cardinality facets

• Maintain segregation across Technical, Business, and Audit observability models

Seniority

• Lead

Nice to have

• Familiarity with AWS Bedrock AgentCore observability (Transaction Search, GenAI Observability)

• Background in Kinesis Firehose metric / log forwarding pipelines (CloudWatch → New Relic)

• Knowledge of cost / token attribution modeling for GenAI workloads

• Skills in Crossplane-based provisioning of observability resources