LLM Analytics Lead

Organization Overview 

Pew Research Center is a nonpartisan, nonadvocacy fact tank that informs the public about the issues, attitudes and trends shaping the world. It does not take policy positions. The Center conducts public opinion polling, demographic research, computational social science research and other data-driven research. It studies politics and policy; news habits and media; the internet and technology; religion; race and ethnicity; international affairs; social, demographic and economic trends; science; research methodology and data science; and immigration and migration. Pew Research Center is a subsidiary of The Pew Charitable Trusts, its primary funder. The Center’s work is carried out by a staff of 180.  

Position Summary: 

The LLM Analytics Lead is a member of Pew Research Center’s digital team. One of the main goals of this position will be to assess the visibility of Pew Research Center’s brand and content across tools like ChatGPT, Gemini, Claude, Perplexity, as well as Google’s AI Overviews/AI Mode. For example, when users ask LLMs questions about public opinion on climate change, how often is our data cited in the results? Another key goal is to explore ways to evaluate and quantify the accuracy and tone with which our work is summarized by these platforms. Based on those learnings, the analytics lead will develop best practices, suggest content experiments and recommend new publishing techniques for the team to improve the visibility and accuracy of our content on external LLMs. 

This is a term-limited, grant-funded position expected to extend 15 months.   
  

Primary Responsibilities 

Collaborate with Center staff to identify key metrics and create a centralized LLM metrics plan. 

Collect and analyze data from specialized LLM visibility metrics tools such as Semrush AIO and Profound, as well as traditional web analytics platforms like Google Analytics, Google Search Console, and Parse.ly. 

Design, build, and refine automated processes to evaluate the accuracy and tone of public LLM chatbot outputs that cite the Center’s work. 

Propose experimental changes to our website content—including structure, metadata, and language—that may influence our performance for specific topics.  

Work with Center staff to design, conduct—and evaluate the results of—content publishing experiments. 

Communicate findings by creating reports and dashboards, giving presentations, and meeting with stakeholders 

Plan and facilitate manual review of LLM output as needed 

Create and oversee the overall project plan for this initiative. 

Education/Training/Experience 

A master’s degree or higher in Data Science, Computer Science, Informatics, or a similar discipline with a focus in natural language processing or artificial intelligence. A bachelor’s degree in one of these fields, combined with significant independent research experience in an academic or non-academic setting, will also be considered. 

1-3 years of experience with data analysis, including synthesis of multiple data sources. 

1–3 years of self-directed research experience in LLM-related benchmarking, with an emphasis on developing new benchmarks rather than optimizing or training models to perform better on existing ones. 

Significant experience developing automated processes to query or collect LLM outputs for analysis, along with conducting manual reviews of those outputs. 

1-3 years of experience working with LLM APIs via Python or R.  

Knowledge, Skill, and Workplace 

Advanced technological skills in data analysis, query design, and coding (Python, R, etc.) 

Solid understanding of the underlying logic of how LLMs work, including concepts from “explainable AI” that clarify why models make particular choices. 

Strong, focused curiosity about LLMs, with the ability to stay current on the rapid evolution of public LLM platforms and chatbot technologies. 

Creative thinking and effective problem-solving skills. 

Maturity and exceptional communication abilities, especially active listening. 

Knowledge of general project management practices, including meeting scheduling and facilitating collaboration with senior, subject-matter experts. 

Familiarity with WordPress CMS (preferred). 

Familiarity with LLM development platforms such as Latitude is a plus 

 
FLSA Status: Exempt  

 
Typical starting salary:  $87,000

Location 

Pew Research Center staff are required to be present in the Center’s Washington, D.C., office three core days weekly (Tuesday, Wednesday, Thursday). Staff may work virtually from remote locations on other days in a typical work week. 

Application Procedure  

Click on the Apply button, and complete required fields. Both cover letter and resume are required.  

When requested, please upload a copy of your resume/cv, as well as a copy of your cover letter in the section labeled Resume/Cover Letter. If the documents have successfully uploaded, you should see two attached files beneath the “Drop files here” box. Please make sure you have uploaded a resume AND a cover letter before moving on to the next page