Artificial Intelligence Researcher/Data Engineer

Hardship Level (not applicable for home-based)

H (no hardship)

Family Type (not applicable for home-based)

Family

Staff Member / Affiliate Type

CONS Local

Target Start Date

2024-07-15

Job Posting End Date

June 30, 2024

Terms of Reference

1. General Background
(Brief description of the national, sector-specific or other relevant context in which the individual contractor will operate).

This is a home based consultancy. The candidate should work from his/her home country or country of residence.

About UNHCR:
UNHCR is a global organisation dedicated to saving lives, protecting rights and building a better future for refugees, forcibly displaced communities and stateless people. Every year, millions of men, women and children are forced to flee their homes to escape conflict and persecution. We are in over 125 countries, using our expertise to protect and care for millions.

About UNHCR Innovation:
UNHCR Innovation Service is a multi-year, inter-departmental initiative looking for more efficient,
effective and creative solutions to challenges that refugees face across UNHCR operations
worldwide. The ultimate goal of the Service is to ensure persons of concern (PoC) protection, self-reliance, empowerment and dignity. The UNHCR Innovation Services does this by rethinking the way UNHCR works, the way it involves PoC in the design of the programs, and the way it looks at good practices in the humanitarian sector. UNHCR Innovation Service, aims to support a culture of
creativity and collaboration across the UN Refugee Agency (UNHCR). UNHCR Innovation serves as
facilitator by bringing innovation tools, and learning environments to those who require support, to
help contribute to improvements in UNHCR work.

About RiMAP:

RiMAP is a UNHCR’s web-based Rights Mapping and Analysis Platform, designed to provide colleagues and partners with an understanding of the legal landscape of rights for refugees, asylum-seekers, internally displaced people, and stateless persons. At its core, it aims to map and analyse existing domestic laws and policies on access to rights for displaced and stateless persons and identify the current protection gaps both in law and practice not only where UNHCR is present in 138 operations but covering, all 193 UN member states and another 10-20 countries/territories.. RiMAP will facilitate national, regional and cross-regional analysis, advance UNHCR’s supervisory role and reinforce the impact of laws and policy work for populations it serves.

The RiMAP project is underway with the core platform scheduled for global availability in the second half of 2024. So while outside the scope of this proposal for 2024, an overview of what is in progress is useful to understand the context, noting that we will continue to build upon skills attained, lessons learned and ensure that best practices are shared widely.

The RiMAP team has already identified use-cases for Generative AI and is currently implementing ‘ad hoc’ solutions to help with accelerating pre-population of the RiMAP platform with standardized data on law and policy, and reducing the burden on country operations – for example:
• Entity extraction - identification of and extraction of law and policy references and other entities from existing documents into a structured form.
• Tagging/categorization – of existing content/“paragraphs” of laws and policies and other documents to enable linkages to the standardized RiMAP rights-based taxonomies
• Summarization/Synthesis (subject to expert human review) of components of law/policy/rights analysis
• Identification of secondary sources and references - laws and policies (and linkages), guidance, analyses

All of the above combined with recognition of the necessary human analysis and human review/approval processes.


2. Purpose and Scope of Assignment
(Concise and detailed description of activities, tasks and responsibilities to be undertaken, including expected travel, if applicable)

The Artificial Intelligence Researcher Engineer/ Data engineer works at the intersection of artificial intelligence, machine-learning, and law and policy analysis. The main goal, working closely with subject-matter experts (legal experts in international and domestic refugee law and policy) is to research approaches to developing an internal “Virtual Legal assistant” for UNHCR staff and by the end of the project to have developed a prototype for further extensive testing before a business decision will be made whether a production version should be developed.

The domain is a fast moving space and the direction of the project may change depending on numerous factors including the rapidly evolving state of the art in GenAI, availability, pricing of models and platforms and most importantly the quality of outputs from such initiatives. The research and development will be in stages where each stage ends with a short evidence-backed report on findings of that stage and delivery of recommendations for the direction of work in subsequent stages.

Some programming/code-writing is essential but the general guidance (unless the research indicates otherwise) would be to choose existing platforms wherever possible rather than writing major components from scratch. It is not expected that the Artificial Intelligence Researcher/Data engineer will develop a full LLMOps stack or create a user interface (UI) or conduct full stack developer dutis.. Instead, the engineer will support in co-designing ways to leverage existing platforms, either commercial or open-source, and make necessary modifications if the platform is open-source. A crucial part of the initial research is to conduct research on the landscape of available tools and recommend the most suitable ones.

While a simple UI is necessary for collecting user feedback, it is not the focus of the project. Options for platforms for logging traces, evaluation include commercial offerings or open-source alternatives such as portkey, promptfoo,promptflow, langfuse, to mention some. However, knowledge of MS Azure's AI stack would be of great assistance to the project. The engineer's role will be to navigate these considerations and make informed decisions that best serve the project's goals.

Deliverables :

Literature Review:
• Deliver a report in MS Word on the main findings, 6-10 pages together with preliminary recommendations for shortlisted platforms/tools and the key requirements to be identified in the next phase

Identification of users, stakeholders and priority needs and data sources
• Working with RiMAP team, document data sources, provisional outcomes/goals, and high-level requirements
• Informal Sensitisation/ Socialisation demos of AI approaches (including RAG) with selected stakeholders using sample documents/corpora (to show benefits and potential pitfalls)
• Refine goals, outcomes and key requirements and deliver a report in MS Word (~5 pages) on recommended priorities and project scope given the time/resources available

Identify criteria for human evaluation of AI outputs for law and policy analysis and demo a proof of concept platform to record evaluations :
• Recommend a platform for measuring/evaluating the quality of outputs from AI/RAG platforms
• Build a demo/proof of concept of the evaluation framework for human feedback (and if possible automated evaluations)
• Deliver a participatory online workshop with key stakeholders/users (<20) to identify and refine core criteria for evaluating quality of outputs based on the needs of stakeholders
• Deliver a report (~5 pages) with results so far and recommendations for direction of further work.

Refine evaluation framework and platform
• Further work to refine the evaluation platform and ensure logging of prompts, RAG processes/pipelines, LLM models for all processes. Ideally the evaluation platform should be able to implemented as a standalone component
• Outputs from platform should include basic reports on perceived quality of combinations of the various components (and indication of costs)
• Legal analysis “prompts” to be continually refined in collaboration with RiMAP team based on results and testing with ‘real-world’ public documents
• Source code to be shared throughout via UNHCR-approved repositories

Documentation, reflective learning and recommendation
• Based on the work so far a draft report ~5 pages summarizing findings, potential pitfalls, potential for value added and provisional recommendations for approach to development of a prototype of a Virtual Legal Assistant.
• Produce a draft of proposed high level requirements, wireframes and workflow for a prototype of a Virtual Legal Assistant.

Validation of recommendations with key stakeholders
• Online participatory workshop to present the drafts in the previous step, recommendations and consultations with Innovation team, RiMAP team and other stakeholders.
• Update and refine the drafts of the report and high level business requirements documents

Development of Prototype Virtual Legal Assistant
• Formal development work of the VLA (unless previous phase recommended further research).

Final report
Final analytical report adapted with feedback provided by UNHCR teams. As extensive as needed but with a summary of the project as a whole, key learnings, recommendations for future work, assessment of success or otherwise, limitations and recommended use cases, in other work etc.

3. Monitoring and Progress Controls
• Weekly progress meetings with RiMAP team
• Milestones as set out in the table above


4. Qualifications and Experience

a. Education
Advanced university degree (Master’s or equivalent) in a field related to computer science, data science, artificial intelligence, human interaction systems, information technologies, software development or any STEM-related area as well as other disciplines with strong data analytics components.

b. Work Experience
Required:
• 2-5 years of experience in working in data science, computer science, cognitive systems, software engineering, information technology, and/or innovation
• At least 1 year in working or conducting applied research with Generative AI or other areas of artificial intelligence or natural language processing.

c. Key Competencies

Required:
• Strong interest in and exposure to humanitarian and development issues, especially in the area of artificial intelligence, machine-learning as well as international law and human rights;
• Proven experience of in software development with Python preferred (e.g. Python, Javascript, PHP, HTML/CSS, REST APIs);
• Working knowledge of the use of LLM GenAI platforms (such as OpenAI, Anthropic, Google Gemini and/or others)
• Working knowledge on various database technologies (e.g. MySQL, SQLServer)
• Demonstrated experience implementing new projects and systems within large organizations;

Desirable experience/knowledge
• Working knowledge of open source or commercial AI agent orchestration platforms (eg MS Autogen, CrewAI, Langchain/LangGraph)
• Experience with MS Azure AI Platform
• Experience with setting up evaluation criteria and methodologies/platforms for human feedback or machine evaluation (or combined) for prompts and results
• Knowledge and ideally implementation of evaluation methodologies for RAG and/or LLM pipelines (eg RAGAS, Langfuse) .
• Experience with the use of GenAI/LLMs in analysis of text-based corpora such as law& policy or corporate knowledge bases
• Experience with open source or commercial vector stores (eg Weaviate, Pinecone, PostgreSQL/pgvector, MS Semantic Search)
• Knowledge of traditional text-based search platforms (Eg Elastic, Lucene etc)
• Experience in open source or commercial semantic segmentation (chunking) of documents (eg Unstructured.io, AI21, LLMSherpa/Tika, Grobid)
• Knowledge of RAG (Retrieval Augmented Generation) concepts, current state of the art and recommended approaches
• Proven experience in documenting research and presenting evidence-based recommendations.

UNHCR competencies
• Innovation and Creativity – Generates innovative and creative solutions to problems that contribute to Organizational and inter-agency effectiveness. Tries different and novel ways of approaching work to create added value in the services provided.
• Knowledge sharing – Actively develops and shares knowledge, leverages expertise to improve organizational and inter-agency effectiveness and add value within sphere of influence.
• Collaboration and Influence – Encourages teamwork and cooperates with others to achieve mutually beneficial results. Gains the support of others for own ideas and initiatives, regardless of formal authority and organizational boundaries.
• Critical Thinking and Judgment – Ability to conceptualize, plan and execute innovative ideas

 

Standard Job Description

Required Languages

,

,

Desired Languages

,

,

Additional Qualifications

Skills

Education

Certifications

Work Experience

Other information

Home based ConsultantThis position doesn't require a functional clearance

Home-Based

Yes