Senior AI Evaluation Specialist - IP Guardrails and Agentic Workflows
Adobe Inc., San Jose, CA, United States
Adobe is redefining creativity through generative AI. Our Firefly family of models powers commercially safe content generation across Creative Cloud, Experience Cloud, and Adobe Express, reaching millions of customers worldwide. As we expand into novel AI capabilities including agentic workflows, multi-model orchestration, and autonomous creative agents, we need a leader who can scale the evaluation and safety guardrails that protect our customers, our brand, and creative communities.
What You\'ll Do IP Safety Evaluation Leadership
Own the end-to-end evaluation strategy for intellectual property safety across all Firefly models and surfaces, including image, video, audio, vector, and 3D generation.
Design, implement, and iterate on scalable evaluation frameworks that detect potential IP infringement - including copyrighted works, trademarks, recognizable characters, brand logos, and likeness/publicity rights.
Partner with Adobe\'s Legal, AI Ethics, and Content Authenticity teams to translate legal and policy requirements into measurable evaluation criteria and test suites.
Establish benchmarks and quality gates that generative models must pass before shipping, maintaining Adobe\'s industry-leading IP indemnification commitment.
Extending to Novel AI Feature Evaluation
Architect evaluation methodologies for emerging AI capabilities beyond image generation - including agentic workflows, autonomous multi-step creative pipelines, AI-powered content orchestration, and agent-to-agent interactions.
Define safety and quality evaluation criteria for agentic systems: scope adherence, action boundary enforcement, hallucination detection, unintended side effect monitoring, and graceful failure modes.
Build proactive red-teaming and adversarial testing programs that stress-test new AI features for misuse, prompt injection, and safety edge cases before they reach customers.
Collaborate with Product, Engineering, and Research to embed evaluation checkpoints into the AI feature development lifecycle - from prototype through GA release.
Develop and maintain dashboards, scorecards, and reporting systems that provide executive insight into AI safety posture across the Firefly portfolio.
What You\'ll Need
5+ years of experience in AI/ML evaluation, trust & safety, content moderation systems, responsible AI, or a closely related technical domain.
Deep expertise in evaluation methodology design: benchmark creation, annotation frameworks, inter-rater reliability, precision/recall analysis, and failure-mode taxonomies.
Strong working knowledge of intellectual property concepts as they apply to generative AI – including copyright, trademark, likeness rights, and fair use considerations.
Experience with multimodal ML systems (vision, language, audio, or video) and an understanding of how generative model architectures produce outputs.
Demonstrated ability to partner with legal, policy, and product teams and translate complex regulatory or legal requirements into actionable technical evaluation plans.
Excellent communication skills with the ability to present technical risk assessments to executive leadership and cross-functional partners.
MS or PhD or equivalent experience in Computer Science, Machine Learning, Information Science, or a related field (or equivalent practical experience).
What Sets You Apart
Experience building evaluation or safety systems specifically for generative AI (image, video, or multimodal models).
Hands-on experience with AI agent evaluation, agentic workflow safety testing, or autonomous system verification.
Background in large-scale data annotation operations, human-in-the-loop evaluation pipelines, or dataset curation for safety.
Familiarity with content authenticity technologies such as C2PA, Content Credentials, or digital provenance systems.
Familiarity with regulatory frameworks relevant to AI safety (EU AI Act, NIST AI RMF, ISO/IEC 42001).
Research contributions or publications in responsible AI, AI safety, fairness, or IP protection in generative systems.
Prior experience at a creative tools company, media organization, or platform with IP-sensitive content at scale.
Adobe is proud to be an Equal Employment Opportunity employer. We do not discriminate based on gender, race or color, ethnicity or national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, or any other protected characteristic.
Expected Pay Range The U.S. pay range for this position is $154,900 – $281,000 annually. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience. Your recruiter can share more about the specific salary range for the job location during the hiring process.
In California, the pay range for this position is $194,000 – $281,000.
In New York, the pay range for this position is $194,000 – $281,000.
In Washington, the pay range for this position is $185,300 – $268,250.
State-Specific Notices California Adobe will consider qualified applicants with arrest or conviction records for employment in accordance with state and local laws and “fair chance” ordinances.
Colorado If this role is open to hiring in Colorado, the application window will remain open until at least the date and time stated above in Pacific Time, in compliance with Colorado pay transparency regulations. If this role does not have Colorado listed as a hiring location, no specific application window applies, and the posting may close at any time based on hiring needs.
Massachusetts It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.
#J-18808-Ljbffr
What You\'ll Do IP Safety Evaluation Leadership
Own the end-to-end evaluation strategy for intellectual property safety across all Firefly models and surfaces, including image, video, audio, vector, and 3D generation.
Design, implement, and iterate on scalable evaluation frameworks that detect potential IP infringement - including copyrighted works, trademarks, recognizable characters, brand logos, and likeness/publicity rights.
Partner with Adobe\'s Legal, AI Ethics, and Content Authenticity teams to translate legal and policy requirements into measurable evaluation criteria and test suites.
Establish benchmarks and quality gates that generative models must pass before shipping, maintaining Adobe\'s industry-leading IP indemnification commitment.
Extending to Novel AI Feature Evaluation
Architect evaluation methodologies for emerging AI capabilities beyond image generation - including agentic workflows, autonomous multi-step creative pipelines, AI-powered content orchestration, and agent-to-agent interactions.
Define safety and quality evaluation criteria for agentic systems: scope adherence, action boundary enforcement, hallucination detection, unintended side effect monitoring, and graceful failure modes.
Build proactive red-teaming and adversarial testing programs that stress-test new AI features for misuse, prompt injection, and safety edge cases before they reach customers.
Collaborate with Product, Engineering, and Research to embed evaluation checkpoints into the AI feature development lifecycle - from prototype through GA release.
Develop and maintain dashboards, scorecards, and reporting systems that provide executive insight into AI safety posture across the Firefly portfolio.
What You\'ll Need
5+ years of experience in AI/ML evaluation, trust & safety, content moderation systems, responsible AI, or a closely related technical domain.
Deep expertise in evaluation methodology design: benchmark creation, annotation frameworks, inter-rater reliability, precision/recall analysis, and failure-mode taxonomies.
Strong working knowledge of intellectual property concepts as they apply to generative AI – including copyright, trademark, likeness rights, and fair use considerations.
Experience with multimodal ML systems (vision, language, audio, or video) and an understanding of how generative model architectures produce outputs.
Demonstrated ability to partner with legal, policy, and product teams and translate complex regulatory or legal requirements into actionable technical evaluation plans.
Excellent communication skills with the ability to present technical risk assessments to executive leadership and cross-functional partners.
MS or PhD or equivalent experience in Computer Science, Machine Learning, Information Science, or a related field (or equivalent practical experience).
What Sets You Apart
Experience building evaluation or safety systems specifically for generative AI (image, video, or multimodal models).
Hands-on experience with AI agent evaluation, agentic workflow safety testing, or autonomous system verification.
Background in large-scale data annotation operations, human-in-the-loop evaluation pipelines, or dataset curation for safety.
Familiarity with content authenticity technologies such as C2PA, Content Credentials, or digital provenance systems.
Familiarity with regulatory frameworks relevant to AI safety (EU AI Act, NIST AI RMF, ISO/IEC 42001).
Research contributions or publications in responsible AI, AI safety, fairness, or IP protection in generative systems.
Prior experience at a creative tools company, media organization, or platform with IP-sensitive content at scale.
Adobe is proud to be an Equal Employment Opportunity employer. We do not discriminate based on gender, race or color, ethnicity or national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, or any other protected characteristic.
Expected Pay Range The U.S. pay range for this position is $154,900 – $281,000 annually. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience. Your recruiter can share more about the specific salary range for the job location during the hiring process.
In California, the pay range for this position is $194,000 – $281,000.
In New York, the pay range for this position is $194,000 – $281,000.
In Washington, the pay range for this position is $185,300 – $268,250.
State-Specific Notices California Adobe will consider qualified applicants with arrest or conviction records for employment in accordance with state and local laws and “fair chance” ordinances.
Colorado If this role is open to hiring in Colorado, the application window will remain open until at least the date and time stated above in Pacific Time, in compliance with Colorado pay transparency regulations. If this role does not have Colorado listed as a hiring location, no specific application window applies, and the posting may close at any time based on hiring needs.
Massachusetts It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.
#J-18808-Ljbffr