Question 1
What is the most suitable library for building a multi-step LLM-based workflow?
- A. Pandas
- B. TensorFlow
- C. PySpark
- D. LangChain
Answer:
D
Explanation:
Problem Context: The Generative AI Engineer needs a tool to build a multi-step LLM-based workflow.
This type of workflow often involves chaining multiple steps together, such as query generation,
retrieval of information, response generation, and post-processing, with LLMs integrated at several
points.
Explanation of Options:
Option A: Pandas: Pandas is a powerful data manipulation library for structured data analysis, but it is
not designed for managing or orchestrating multi-step workflows, especially those involving LLMs.
Option B: TensorFlow: TensorFlow is primarily used for training and deploying machine learning
models, especially deep learning models. It is not designed for orchestrating multi-step tasks in LLM-
based workflows.
Option C: PySpark: PySpark is a distributed computing framework used for large-scale data
processing. While useful for handling big data, it is not specialized for chaining LLM-based
operations.
Option D: LangChain: LangChain is a purpose-built framework designed specifically for orchestrating
multi-step workflows with large language models (LLMs). It enables developers to easily chain
different tasks, such as retrieving documents, summarizing information, and generating responses,
all in a structured flow. This makes it the best tool for building complex LLM-based workflows.
Thus, LangChain is the most suitable library for creating multi-step LLM-based workflows.
Comments
Question 2
When developing an LLM application, it’s crucial to ensure that the data used for training the model
complies with licensing requirements to avoid legal risks.
Which action is NOT appropriate to avoid legal risks?
- A. Reach out to the data curators directly before you have started using the trained model to let them know.
- B. Use any available data you personally created which is completely original and you can decide what license to use.
- C. Only use data explicitly labeled with an open license and ensure the license terms are followed.
- D. Reach out to the data curators directly after you have started using the trained model to let them know.
Answer:
D
Explanation:
Problem Context: When using data to train a model, it’s essential to ensure compliance with
licensing to avoid legal risks. Legal issues can arise from using data without permission, especially
when it comes from third-party sources.
Explanation of Options:
Option A: Reaching out to data curators before using the data is an appropriate action. This allows
you to ensure you have permission or understand the licensing terms before starting to use the data
in your model.
Option B: Using original data that you personally created is always a safe option. Since you have full
ownership over the data, there are no legal risks, as you control the licensing.
Option C: Using data that is explicitly labeled with an open license and adhering to the license terms
is a correct and recommended approach. This ensures compliance with legal requirements.
Option D: Reaching out to the data curators after you have already started using the trained model is
not appropriate. If you’ve already used the data without understanding its licensing terms, you may
have already violated the terms of use, which could lead to legal complications. It’s essential to
clarify the licensing terms before using the data, not after.
Thus, Option D is not appropriate because it could expose you to legal risks by using the data without
first obtaining the proper licensing permissions.
Comments
Question 3
A Generative AI Engineer is testing a simple prompt template in LangChain using the code below, but
is getting an error.
Assuming the API key was properly defined, what change does the Generative AI Engineer need to
make to fix their chain?
A)
B)
C)
D)
- A. Option A
- B. Option B
- C. Option C
- D. Option D
Answer:
C
Explanation:
To fix the error in the LangChain code provided for using a simple prompt template, the correct
approach is Option C. Here's a detailed breakdown of why Option C is the right choice and how it
addresses the issue:
Proper Initialization: In Option C, the LLMChain is correctly initialized with the LLM instance specified
as OpenAI(), which likely represents a language model (like GPT) from OpenAI. This is crucial as it
specifies which model to use for generating responses.
Correct Use of Classes and Methods:
The PromptTemplate is defined with the correct format, specifying that adjective is a variable within
the template. This allows dynamic insertion of values into the template when generating text.
The prompt variable is properly linked with the PromptTemplate, and the final template string is
passed correctly.
The LLMChain correctly references the prompt and the initialized OpenAI() instance, ensuring that
the template and the model are properly linked for generating output.
Why Other Options Are Incorrect:
Option A: Misuses the parameter passing in generate method by incorrectly structuring the
dictionary.
Option B: Incorrectly uses prompt.format method which does not exist in the context of LLMChain
and PromptTemplate configuration, resulting in potential errors.
Option D: Incorrect order and setup in the initialization parameters for LLMChain, which would likely
lead to a failure in recognizing the correct configuration for prompt and LLM usage.
Thus, Option C is correct because it ensures that the LangChain components are correctly set up and
integrated, adhering to proper syntax and logical flow required by LangChain's architecture. This
setup avoids common pitfalls such as type errors or method misuses, which are evident in other
options.
Comments
Question 4
A Generative Al Engineer is creating an LLM system that will retrieve news articles from the year
1918 and related to a user's query and summarize them. The engineer has noticed that the
summaries are generated well but often also include an explanation of how the summary was
generated, which is undesirable.
Which change could the Generative Al Engineer perform to mitigate this issue?
- A. Split the LLM output by newline characters to truncate away the summarization explanation.
- B. Tune the chunk size of news articles or experiment with different embedding models.
- C. Revisit their document ingestion logic, ensuring that the news articles are being ingested properly.
- D. Provide few shot examples of desired output format to the system and/or user prompt.
Answer:
D
Explanation:
To mitigate the issue of the LLM including explanations of how summaries are generated in its
output, the best approach is to adjust the training or prompt structure. Here’s why Option D is
effective:
Few-shot Learning: By providing specific examples of how the desired output should look (i.e., just
the summary without explanation), the model learns the preferred format. This few-shot learning
approach helps the model understand not only what content to generate but also how to format its
responses.
Prompt Engineering: Adjusting the user prompt to specify the desired output format clearly can
guide the LLM to produce summaries without additional explanatory text. Effective prompt design is
crucial in controlling the behavior of generative models.
Why Other Options Are Less Suitable:
A: While technically feasible, splitting the output by newline and truncating could lead to loss of
important content or create awkward breaks in the summary.
B: Tuning chunk sizes or changing embedding models does not directly address the issue of the
model's tendency to generate explanations along with summaries.
C: Revisiting document ingestion logic ensures accurate source data but does not influence how the
model formats its output.
By using few-shot examples and refining the prompt, the engineer directly influences the output
format, making this approach the most targeted and effective solution.
Comments
Question 5
A Generative Al Engineer has developed an LLM application to answer questions about internal
company policies. The Generative AI Engineer must ensure that the application doesn’t hallucinate
or leak confidential data.
Which approach should NOT be used to mitigate hallucination or confidential data leakage?
- A. Add guardrails to filter outputs from the LLM before it is shown to the user
- B. Fine-tune the model on your data, hoping it will learn what is appropriate and not
- C. Limit the data available based on the user’s access level
- D. Use a strong system prompt to ensure the model aligns with your needs.
Answer:
B
Explanation:
When addressing concerns of hallucination and data leakage in an LLM application for internal
company policies, fine-tuning the model on internal data with the hope it learns data boundaries can
be problematic:
Risk of Data Leakage: Fine-tuning on sensitive or confidential data does not guarantee that the model
will not inadvertently include or reference this data in its outputs. There’s a risk of overfitting to the
specific data details, which might lead to unintended leakage.
Hallucination: Fine-tuning does not necessarily mitigate the model's tendency to hallucinate; in fact,
it might exacerbate it if the training data is not comprehensive or representative of all potential
queries.
Better Approaches:
A, C, and D involve setting up operational safeguards and constraints that directly address data
leakage and ensure responses are aligned with specific user needs and security levels.
Fine-tuning lacks the targeted control needed for such sensitive applications and can introduce new
risks, making it an unsuitable approach in this context.
Comments
Question 6
A Generative Al Engineer interfaces with an LLM with prompt/response behavior that has been
trained on customer calls inquiring about product availability. The LLM is designed to output “In
Stock” if the product is available or only the term “Out of Stock” if not.
Which prompt will work to allow the engineer to respond to call classification labels correctly?
- A. Respond with “In Stock” if the customer asks for a product.
- B. You will be given a customer call transcript where the customer asks about product availability. The outputs are either “In Stock” or “Out of Stock”. Format the output in JSON, for example: {“call_id”: “123”, “label”: “In Stock”}.
- C. Respond with “Out of Stock” if the customer asks for a product.
- D. You will be given a customer call transcript where the customer inquires about product availability. Respond with “In Stock” if the product is available or “Out of Stock” if not.
Answer:
B
Explanation:
Problem Context: The Generative AI Engineer needs a prompt that will enable an LLM trained on
customer call transcripts to classify and respond correctly regarding product availability. The desired
response should clearly indicate whether a product is "In Stock" or "Out of Stock," and it should be
formatted in a way that is structured and easy to parse programmatically, such as JSON.
Explanation of Options:
Option A: Respond with “In Stock” if the customer asks for a product. This prompt is too generic and
does not specify how to handle the case when a product is not available, nor does it provide a
structured output format.
Option B: This option is correctly formatted and explicit. It instructs the LLM to respond based on the
availability mentioned in the customer call transcript and to format the response in JSON. This
structure allows for easy integration into systems that may need to process this information
automatically, such as customer service dashboards or databases.
Option C: Respond with “Out of Stock” if the customer asks for a product. Like option A, this prompt
is also insufficient as it only covers the scenario where a product is unavailable and does not provide
a structured output.
Option D: While this prompt correctly specifies how to respond based on product availability, it lacks
the structured output format, making it less suitable for systems that require formatted data for
further processing.
Given the requirements for clear, programmatically usable outputs, Option B is the optimal choice
because it provides precise instructions on how to respond and includes a JSON format example for
structuring the output, which is ideal for automated systems or further data handling.
Comments
Question 7
A Generative Al Engineer is tasked with developing a RAG application that will help a small internal
group of experts at their company answer specific questions, augmented by an internal knowledge
base. They want the best possible quality in the answers, and neither latency nor throughput is a
huge concern given that the user group is small and they’re willing to wait for the best answer. The
topics are sensitive in nature and the data is highly confidential and so, due to regulatory
requirements, none of the information is allowed to be transmitted to third parties.
Which model meets all the Generative Al Engineer’s needs in this situation?
- A. Dolly 1.5B
- B. OpenAI GPT-4
- C. BGE-large
- D. Llama2-70B
Answer:
C
Explanation:
Problem Context: The Generative AI Engineer needs a model for a Retrieval-Augmented Generation
(RAG) application that provides high-quality answers, where latency and throughput are not major
concerns. The key factors are confidentiality and sensitivity of the data, as well as the requirement
for all processing to be confined to internal resources without external data transmission.
Explanation of Options:
Option A: Dolly 1.5B: This model does not typically support RAG applications as it's more focused on
image generation tasks.
Option B: OpenAI GPT-4: While GPT-4 is powerful for generating responses, its standard deployment
involves cloud-based processing, which could violate the confidentiality requirements due to
external data transmission.
Option C: BGE-large: The BGE (Big Green Engine) large model is a suitable choice if it is configured to
operate on-premises or within a secure internal environment that meets regulatory requirements.
Assuming this setup, BGE-large can provide high-quality answers while ensuring that data is not
transmitted to third parties, thus aligning with the project's sensitivity and confidentiality needs.
Option D: Llama2-70B: Similar to GPT-4, unless specifically set up for on-premises use, it generally
relies on cloud-based services, which might risk confidential data exposure.
Given the sensitivity and confidentiality concerns, BGE-large is assumed to be configurable for secure
internal use, making it the optimal choice for this scenario.
Comments
Question 8
A Generative Al Engineer would like an LLM to generate formatted JSON from emails. This will
require parsing and extracting the following information: order ID, date, and sender email. Here’s a
sample email:
They will need to write a prompt that will extract the relevant information in JSON format with the
highest level of output accuracy.
Which prompt will do that?
- A. You will receive customer emails and need to extract date, sender email, and order ID. You should return the date, sender email, and order ID information in JSON format.
- B. You will receive customer emails and need to extract date, sender email, and order ID. Return the extracted information in JSON format. Here’s an example: {“date”: “April 16, 2024”, “sender_email”: “[email protected]”, “order_id”: “RE987D”}
- C. You will receive customer emails and need to extract date, sender email, and order ID. Return the extracted information in a human-readable format.
- D. You will receive customer emails and need to extract date, sender email, and order ID. Return the extracted information in JSON format.
Answer:
B
Explanation:
Problem Context: The goal is to parse emails to extract certain pieces of information and output this
in a structured JSON format. Clarity and specificity in the prompt design will ensure higher accuracy
in the LLM’s responses.
Explanation of Options:
Option A: Provides a general guideline but lacks an example, which helps an LLM understand the
exact format expected.
Option B: Includes a clear instruction and a specific example of the output format. Providing an
example is crucial as it helps set the pattern and format in which the information should be
structured, leading to more accurate results.
Option C: Does not specify that the output should be in JSON format, thus not meeting the
requirement.
Option D: While it correctly asks for JSON format, it lacks an example that would guide the LLM on
how to structure the JSON correctly.
Therefore, Option B is optimal as it not only specifies the required format but also illustrates it with
an example, enhancing the likelihood of accurate extraction and formatting by the LLM.
Comments
Question 9
A Generative AI Engineer has been asked to build an LLM-based question-answering application. The
application should take into account new documents that are frequently published. The engineer
wants to build this application with the least cost and least development effort and have it operate at
the lowest cost possible.
Which combination of chaining components and configuration meets these requirements?
- A. For the application a prompt, a retriever, and an LLM are required. The retriever output is inserted into the prompt which is given to the LLM to generate answers.
- B. The LLM needs to be frequently with the new documents in order to provide most up-to-date answers.
- C. For the question-answering application, prompt engineering and an LLM are required to generate answers.
- D. For the application a prompt, an agent and a fine-tuned LLM are required. The agent is used by the LLM to retrieve relevant content that is inserted into the prompt which is given to the LLM to generate answers.
Answer:
A
Explanation:
Problem Context: The task is to build an LLM-based question-answering application that integrates
new documents frequently with minimal costs and development efforts.
Explanation of Options:
Option A: Utilizes a prompt and a retriever, with the retriever output being fed into the LLM. This
setup is efficient because it dynamically updates the data pool via the retriever, allowing the LLM to
provide up-to-date answers based on the latest documents without needing to frequently retrain the
model. This method offers a balance of cost-effectiveness and functionality.
Option B: Requires frequent retraining of the LLM, which is costly and labor-intensive.
Option C: Only involves prompt engineering and an LLM, which may not adequately handle the
requirement for incorporating new documents unless it’s part of an ongoing retraining or updating
mechanism, which would increase costs.
Option D: Involves an agent and a fine-tuned LLM, which could be overkill and lead to higher
development and operational costs.
Option A is the most suitable as it provides a cost-effective, minimal development approach while
ensuring the application remains up-to-date with new information.
Comments
Question 10
A Generative AI Engineer is creating an agent-based LLM system for their favorite monster truck
team. The system can answer text based questions about the monster truck team, lookup event
dates via an API call, or query tables on the team’s latest standings.
How could the Generative AI Engineer best design these capabilities into their system?
- A. Ingest PDF documents about the monster truck team into a vector store and query it in a RAG architecture.
- B. Write a system prompt for the agent listing available tools and bundle it into an agent system that runs a number of calls to solve a query.
- C. Instruct the LLM to respond with “RAG”, “API”, or “TABLE” depending on the query, then use text parsing and conditional statements to resolve the query.
- D. Build a system prompt with all possible event dates and table information in the system prompt. Use a RAG architecture to lookup generic text questions and otherwise leverage the information in the system prompt.
Answer:
B
Explanation:
In this scenario, the Generative AI Engineer needs to design a system that can handle different types
of queries about the monster truck team. The queries may involve text-based information, API
lookups for event dates, or table queries for standings. The best solution is to implement a tool-
based agent system.
Here’s how option B works, and why it’s the most appropriate answer:
System Design Using Agent-Based Model:
In modern agent-based LLM systems, you can design a system where the LLM (Large Language
Model) acts as a central orchestrator. The model can "decide" which tools to use based on the query.
These tools can include API calls, table lookups, or natural language searches. The system should
contain a system prompt that informs the LLM about the available tools.
System Prompt Listing Tools:
By creating a well-crafted system prompt, the LLM knows which tools are at its disposal. For instance,
one tool may query an external API for event dates, another might look up standings in a database,
and a third may involve searching a vector database for general text-based information. The agent
will be responsible for calling the appropriate tool depending on the query.
Agent Orchestration of Calls:
The agent system is designed to execute a series of steps based on the incoming query. If a user asks
for the next event date, the system will recognize this as a task that requires an API call. If the user
asks about standings, the agent might query the appropriate table in the database. For text-based
questions, it may call a search function over ingested data. The agent orchestrates this entire
process, ensuring the LLM makes calls to the right resources dynamically.
Generative AI Tools and Context:
This is a standard architecture for integrating multiple functionalities into a system where each query
requires different actions. The core design in option B is efficient because it keeps the system
modular and dynamic by leveraging tools rather than overloading the LLM with static information in
a system prompt (like option D).
Why Other Options Are Less Suitable:
A (RAG Architecture): While relevant, simply ingesting PDFs into a vector store only helps with text-
based retrieval. It wouldn’t help with API lookups or table queries.
C (Conditional Logic with RAG/API/TABLE): Although this approach works, it relies heavily on manual
text parsing and might introduce complexity when scaling the system.
D (System Prompt with Event Dates and Standings): Hardcoding dates and table information into a
system prompt isn’t scalable. As the standings or events change, the system would need constant
updating, making it inefficient.
By bundling multiple tools into a single agent-based system (as in option B), the Generative AI
Engineer can best handle the diverse requirements of this system.
Comments
Page 1 out of 6
Viewing questions 1-10 out of 61
page 2