> For the complete documentation index, see [llms.txt](https://theaihandbook.leomohan.net/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theaihandbook.leomohan.net/chapter-14-what-questions-should-i-ask-about-ai-tools-before-using-them.md). # Chapter 14: What Questions Should I Ask About AI Tools Before Using Them? ### The “Buyer’s Guide” Chapter **Q1: What problem am I actually trying to solve?** **A:** This is the most important question, and it’s the one people skip most often. They hear about a cool new AI tool and think “I should use that!” without clarifying what they actually need. **Why this matters:** AI is a solution looking for problems. But not every problem needs an AI solution. Using AI when a simpler tool would work is like using a chainsaw to cut butter—overkill, messy, and likely to create new problems. **Questions to ask yourself:** * What specific task is taking too much time? * What am I trying to achieve that I can’t achieve now? * What’s the pain point I’m trying to address? * Is this a one-time need or something ongoing? **Example:** If you need to send a simple thank-you email, AI is overkill. If you need to write 50 personalized thank-you emails to donors, AI might be perfect. **The risk of skipping this question:** You’ll adopt tools that don’t actually help, waste time learning them, and get frustrated when they don’t solve problems they were never designed for. **Before trying any AI tool, write down:** “The problem I’m trying to solve is \[specific problem]. Currently, I handle it by \[current method]. That takes \[time/effort]. I hope AI can \[specific improvement].” If you can’t complete that sentence, you’re not ready to choose a tool. **Q2: Is this the right tool for the job, or is it overkill?** **A:** Once you know your problem, ask whether this specific AI tool is appropriate—or whether something simpler would work better. **The hierarchy of tools:** **No tool:** For simple tasks, just do it yourself. Writing a two-sentence email? Just type it. **Basic automation:** If-then rules, templates, macros. For repetitive but predictable tasks, these are simpler and more reliable than AI. **Specialized AI:** Tools designed for specific tasks (Grammarly for writing, Otter for transcription). These often work better than general AI for their niche. **General AI:** ChatGPT, Claude, etc. For tasks that require flexibility and creativity, these are great. But they’re also more complex and less predictable. **The overkill test:** * Does this task require creativity or just consistency? * Does it need to be done once or repeatedly? * How much would it cost (time, money, learning) to use AI? * What’s the cost of AI making a mistake? **Example:** You need to extract data from 100 PDF invoices. A specialized tool might do this perfectly. ChatGPT might also do it, but might hallucinate numbers. The specialized tool is the right choice; ChatGPT is overkill with more risk. **The principle:** Use the simplest tool that solves your problem. AI is powerful but complex. Don’t use it when simpler tools suffice. **Q3: What data does this AI collect about me?** **A:** Before using any AI tool, understand what information you’re giving away. Many free AI services are “free” because your data is the product. **Questions to investigate:** * Does the tool save my conversations? * Can human reviewers see my data? * Is my data used to train future models? * Where is my data stored (which country)? * How long is my data retained? * Can I delete my data? * Is my data shared with third parties? **Where to find answers:** Privacy policy (yes, read it or at least scan for these points). Terms of service. Often there’s a “Privacy” section on the website. **Red flags:** * Vague language about data use * No clear deletion options * Data used for training by default (opt-out may be hidden) * Data shared with “partners” (undefined) * Different standards for free vs. paid users **What this means for you:** * Don’t share personal information you wouldn’t want public * Assume anything you type could be seen by humans * For sensitive work, consider paid tools with stronger privacy guarantees * Some industries (healthcare, law, finance) have specific compliance requirements **The trade-off:** Free tools often monetize your data. Paid tools often promise more privacy. Decide what your privacy is worth. **Q4: Where is my data stored and who can see it?** **A:** Data storage location matters for legal and security reasons. Different countries have different privacy laws, and different companies have different security practices. **Key considerations:** **Jurisdiction:** If your data is stored in the US, it’s subject to US laws (including surveillance laws like the Patriot Act). If stored in Europe, it’s subject to GDPR. If stored in China, different rules apply. **Company access:** Can employees of the AI company see your data? For customer support, training, or quality improvement? Some companies use human reviewers to improve their models—meaning actual people might read your conversations. **Government access:** In some countries, governments can demand access to data. If you’re discussing sensitive topics, this matters. **Third-party access:** Does the company share data with partners, advertisers, or other third parties? **What to look for:** * Clear statements about data residency * Transparency about employee access * No hidden third-party sharing * Compliance with relevant regulations (GDPR, HIPAA, etc.) **For sensitive work:** Consider tools that offer: * On-device processing (data never leaves your device) * Private deployments (runs on your company’s servers) * End-to-end encryption * No human review of your data **The bottom line:** If you wouldn’t want it on the front page of a newspaper, don’t put it in an AI tool unless you’ve verified exactly who can see it. **Q5: Can I trust this company with my information?** **A:** Trust is earned through transparency, history, and practices. Before committing to an AI tool, evaluate the company behind it. **Questions to research:** **Reputation:** * How long has the company existed? * What’s their reputation in the industry? * Have they had data breaches before? * What do users say about them? **Business model:** * How do they make money? * If it’s free, you’re likely the product * If you pay, what does that cover? * Are they financially stable? **Transparency:** * Do they clearly explain their data practices? * Are they responsive to user concerns? * Do they have a clear privacy policy? * Have they made public commitments about responsible AI? **Security practices:** * Do they offer two-factor authentication? * How do they protect data? * Have they been audited by third parties? **Track record:** * Have they had incidents and how did they handle them? * Do they communicate openly about problems? * Have they made commitments they later broke? **Red flags:** * Vague or evasive answers to privacy questions * Recent controversies about data misuse * Unclear ownership or funding * No clear way to contact support **The principle:** Trust is built over time. For important or sensitive use, established companies with clear practices are safer than new, unknown tools—even if the new tools have flashier features. **Q6: How accurate is this AI for my specific use case?** **A:** General claims of accuracy are meaningless. What matters is accuracy for *your specific task*with *your specific data*. **Why general claims mislead:** A company might say “95% accurate” based on their test data. But: * Their test data may not match your use case * “Accuracy” might mean different things for different tasks * Benchmarks are often chosen to make the company look good **How to evaluate real accuracy:** **Test it yourself:** Run your own tests with realistic examples. Don’t just trust marketing claims. **Use representative data:** Test with the kind of inputs you’ll actually use. If you’re analyzing legal documents, test with legal documents, not generic text. **Check edge cases:** How does it perform on unusual inputs? On the boundaries of what you need? **Look for independent evaluations:** Are there third-party benchmarks? User reviews? Academic studies? **Consider the cost of error:** For a recipe generator, errors are annoying but harmless. For medical advice, errors could be fatal. Your tolerance depends on consequences. **Ask specific questions:** * “How does this perform on \[your specific task]?” * “What are its known limitations?” * “What kind of mistakes does it typically make?” **The bottom line:** Trust but verify. Test before you commit. And never assume AI accuracy without evidence relevant to your use case. **Q7: What happens when this AI makes a mistake?** **A:** AI will make mistakes. The question isn’t if, but when—and what happens next. This is one of the most important questions to ask before relying on any AI tool. **Questions to consider:** **Detection:** * How will I know when it makes a mistake? * Does the tool flag uncertainty? * Are errors obvious or subtle? **Impact:** * What’s the worst that could happen if it’s wrong? * Who is affected? * Can mistakes be corrected before causing harm? **Accountability:** * Who is responsible for errors—you or the company? * What does the terms of service say about liability? * (Spoiler: They almost always disclaim all liability.) **Recovery:** * Can errors be fixed easily? * Is there a way to appeal or correct? * How much time/effort does recovery take? **Real-world example:** A lawyer used AI to write a legal brief. The AI invented fake cases with real-sounding citations. The lawyer filed it without checking. The judge was not amused. The lawyer faced sanctions. The AI company’s terms said “not liable.” **What to do:** * Always verify important AI outputs * Have a backup plan for when AI fails * Don’t use AI for tasks where errors would be catastrophic * Understand that ultimately, you’re responsible for what you produce with AI **The principle:** AI is a tool that amplifies your capabilities—and your responsibility. You can’t delegate accountability to a machine. **Q8: Can I correct or override the AI’s decisions?** **A:** Some AI systems are designed to be collaborative—you can review, edit, and override their outputs. Others are more autonomous, making decisions that are difficult to change. **Questions to ask:** **For generative AI (writing, images, etc.):** * Can I edit the output easily? * Does the tool allow iteration and refinement? * Can I provide feedback that improves results? **For decision-making AI (screening, scoring, etc.):** * Is there a way to appeal decisions? * Can a human override the AI? * How difficult is the override process? * Are there safeguards against automation bias (assuming AI is right)? **For automated systems:** * Can I stop or pause the AI if something goes wrong? * Is there a human review process for important decisions? * How much autonomy does the AI actually have? **Why this matters:** Systems that don’t allow override or correction remove human judgment from the loop. This is fine for low-stakes tasks (spam filtering) but dangerous for consequential decisions (hiring, lending, medical diagnosis). **What to look for:** * Clear override mechanisms * Transparent decision-making * Human review for important cases * The ability to provide feedback and see improvements **The ideal:** AI as an assistant, not an authority. It suggests; you decide. It drafts; you edit. It flags; you investigate. You remain in control. **Q9: How was this AI trained and on what data?** **A:** The training data determines what the AI knows, how it behaves, and what biases it has. Understanding this helps you assess whether it’s right for your needs. **Questions to ask:** **Data sources:** * What data was used for training? * Was it publicly available or proprietary? * Does it include diverse perspectives and sources? * When was the data collected? (Is it up to date?) **Data quality:** * Was the data cleaned and filtered? * How was it labeled (if supervised learning)? * Were there efforts to reduce bias? * What’s missing from the training data? **Transparency:** * Does the company disclose training data sources? * Are there known limitations or biases? * Has the model been independently evaluated? **Why this matters:** * If trained on internet data, it reflects internet biases * If trained mostly on English sources, it has cultural blind spots * If trained on data before 2021, it lacks recent knowledge * If trained on specific domains, it may not generalize **Examples:** A medical AI trained mostly on data from one demographic may not work well for others. A hiring AI trained on past successful hires may perpetuate past biases. **What you can do:** * Look for transparency about training * Test for biases relevant to your use * Supplement with domain-specific knowledge * Don’t assume “trained on everything” means “trained well on everything” **Q10: Is this AI biased in ways that might affect me?** **A:** All AI inherits biases from its training data. The question is whether those biases matter for your specific use case. **Common biases to check for:** **Cultural bias:** Does it understand your cultural context? If you’re not from a Western, English-speaking background, the AI may miss nuances or make incorrect assumptions. **Gender bias:** Does it default to certain genders for certain roles? Does it make assumptions based on names or pronouns? **Racial/ethnic bias:** Does it perform differently for different groups? This is especially important for any tool making decisions about people. **Political bias:** Does it favor certain political perspectives? This matters for content creation, research, or any politically sensitive topic. **Age bias:** Does it favor newer information over older, even when older perspectives are relevant? **Socioeconomic bias:** Does it make assumptions based on income, education, or background? **How to check:** * Test with diverse inputs representing different groups * Ask the same question in different ways * Look for independent bias audits * Read user reviews from people with different backgrounds **What to do if you find bias:** * Be aware of the limitations * Supplement with other sources * Provide counterexamples in your prompts * Consider whether a different tool might be less biased **The bottom line:** No AI is unbiased. The goal is to understand the biases and decide if they’re acceptable for your use. **Q11: How much does it really cost (hidden costs)?** **A:** The listed price is just the beginning. AI tools often come with hidden costs that can add up. **Obvious costs:** * Monthly subscription fees * Per-use or per-token charges * Tiered pricing for more features **Hidden costs:** **Time to learn:** How long will it take to learn this tool effectively? Your time is valuable. **Prompt engineering:** Getting good results takes practice and experimentation. That’s time and effort. **Verification:** You still need to check AI outputs. That takes time, especially for important work. **Integration:** Does it work with your existing tools? If not, you may need to build integrations or change workflows. **Training:** For specialized uses, you may need to fine-tune the model or provide examples. That costs time/money. **Data preparation:** Many AI tools require clean, formatted data. Preparing that data takes work. **Hidden fees:** * Overages when you exceed usage limits * API costs that add up * Premium features that aren’t in base plans * Team or enterprise pricing that’s much higher **Exit costs:** If you want to leave, can you export your data? Is it in a usable format? Switching costs can be significant. **The real calculation:** Total cost = subscription + your time (learning + using + verifying) + integration costs + potential overages Compare this to the value you actually get. Sometimes the free option with more of your time is better. Sometimes paying saves enough time to be worth it. **Q12: Do I need special skills to use this effectively?** **A:** Some AI tools are designed for everyone. Others assume technical expertise. Knowing the difference saves frustration. **Questions to ask:** **User interface:** * Is it point-and-click or command-line? * Does it have good documentation? * Are there tutorials and examples? * Is the interface intuitive? **Technical requirements:** * Do I need to know how to code? * Does it require understanding APIs? * Will I need to set up my own infrastructure? * Are there system requirements (powerful computer, etc.)? **Prompting skill:** Some tools work with simple questions. Others require sophisticated prompt engineering. Where does this one fall? **Domain knowledge:** Do I need expertise in the field to evaluate outputs? For medical or legal AI, absolutely. For recipe ideas, less so. **Learning curve:** * How long to basic competence? * How long to mastery? * Is the learning curve steep or gradual? **Support:** * Is there customer support? * Are there communities of users? * Can I find help when stuck? **Be honest with yourself:** A tool that requires coding skills is useless if you can’t code. A tool that needs prompt engineering is frustrating if you just want simple answers. **The principle:** Choose tools that match your skills—or be prepared to invest in learning. There’s no shame in starting with simpler tools and working up. **Q13: Can I export my data if I want to leave?** **A:** This is the “exit strategy” question. Before committing to any tool, understand how you’d get your data out. **Why this matters:** * You might find a better tool later * The company might raise prices * The tool might shut down * You might need to switch for compliance reasons **Questions to ask:** **Export options:** * Can I export my data? * In what formats (JSON, CSV, plain text, etc.)? * Is export a manual process or automated? * Is there an API to access my data? **Data ownership:** * Who owns the content I create with the tool? * Can I take my creations elsewhere? * Are there restrictions on moving data? **Format lock-in:** Is data stored in a proprietary format that only this tool can read? If so, you’re trapped. **Deletion:** * Can I delete my data when I leave? * Is it actually deleted from their systems? **The nightmare scenario:** You’ve created years of work in a tool. You want to switch. You discover you can’t export your data, or it exports in a format nothing else can read. You’re stuck. **What to do:** * Check export options before committing * For important work, keep backups in standard formats * Consider open standards and formats when possible * Read the fine print about data portability **The principle:** Your data should be yours. A tool that traps your data is a tool to avoid. **Q14: How often is the AI updated and improved?** **A:** AI is moving fast. A tool that’s great today might be obsolete in six months—or might get worse if updates introduce problems. **Questions to ask:** **Update frequency:** * How often are new versions released? * Is there a roadmap of planned improvements? * Do they communicate about changes? **Model versions:** * Can I choose which version to use? * If they update, does performance change? * Are there breaking changes that require me to adapt? **Improvement trajectory:** * Is the tool getting better over time? * Read reviews from different time periods * Check if user complaints are addressed **Stability:** * Do updates introduce bugs? * Is there a testing process before release? * Can I rely on consistent performance? **For critical use:** Some applications need stability over novelty. You might prefer a tool that changes slowly if you need consistent, predictable results. **The trade-off:** Frequent updates mean new features but also potential instability. Infrequent updates mean stability but risk obsolescence. **What to watch for:** * Major version changes that break workflows * “Improvements” that actually make things worse for your use case * Lack of communication about changes * No clear way to provide feedback **The principle:** Choose tools with healthy development practices—regular updates, clear communication, and responsiveness to users. **Q15: Is there human support when things go wrong?** **A:** AI tools fail. When they do, you need help. The availability and quality of human support can make or break your experience. **Questions to ask:** **Support availability:** * Is there customer support? * What channels (email, chat, phone)? * What hours (24/7 or business hours only)? * Is support included or extra cost? **Support quality:** * Do they actually help? * Are support staff knowledgeable? * How long do responses take? * Read reviews about support experiences **Self-help resources:** * Is there documentation? * Are there tutorials and FAQs? * Is there a user community? * Can you find answers without contacting support? **For critical use:** If the tool is essential to your work, you need reliable support. A tool with no support and no community is risky. **The free tool reality:** Free tools often have limited or no support. You’re on your own. That’s fine for experimentation but risky for critical work. **What to do:** * Test support before you need it (ask a question) * Check community forums for activity * Have backup plans for when tools fail * For business use, paid plans usually include better support **The bottom line:** When AI breaks (and it will), who will help? Make sure you have an answer before you’re stuck. **Q16: What do other users say about it?** **A:** Marketing says what the company wants you to believe. Users say what actually happens. Both are valuable, but user reviews are often more honest. **Where to look:** **Review sites:** G2, Capterra, Trustpilot—sites dedicated to software reviews. Look for patterns, not individual opinions. **Social media:** Search for the tool on Twitter, Reddit, LinkedIn. See what real users are saying—both praise and complaints. **Professional communities:** Slack groups, Discord servers, industry forums. These often have honest, detailed discussions. **YouTube:** Video reviews can show you the tool in action, not just screenshots. **What to look for:** **Patterns:** * Multiple people saying the same thing (good or bad) is significant * One-off complaints might be user error * Consistent themes reveal real strengths/weaknesses **Specific use cases:** Find users doing similar things to what you plan. Their experience is most relevant. **Recent reviews:** AI changes fast. Reviews from two years ago may be irrelevant. Focus on recent experiences. **The company’s response:** How do they handle criticism? Do they engage constructively? Defensive or dismissive responses are red flags. **What to watch for:** * Fake reviews (overly positive, generic language) * Review bombing (coordinated negative campaigns) * Astroturfing (fake grassroots support) **The principle:** Trust patterns, not outliers. Look for users like you. Be skeptical of both perfect scores and one-star rants. **Q17: Is the free version good enough or do I need to pay?** **A:** Most AI tools use a freemium model—free version with limits, paid version with more features. Deciding which you need requires honest self-assessment. **What free versions typically offer:** * Limited usage (messages per day, images per month) * Older/weaker models * No priority access (slow during peak times) * Basic features only * Watermarks on generated content * Limited or no support **What paid versions add:** * Higher usage limits * Latest models * Priority access * Advanced features * No watermarks * Support * Privacy protections (data not used for training) **Questions to decide:** **How much will you use it?** Occasional use? Free might be fine. Daily reliance? Paid probably worth it. **How important is quality?** If you need the best results, pay for the latest models. For casual use, free models are often good enough. **How much is your time worth?** Free version might be slower or more limited. If it saves you hours, paid is worth it. **Is privacy important?** Free often means your data trains the model. Paid often promises better privacy. Decide what matters. **Can you live with limits?** Track your usage. If you regularly hit limits, paid makes sense. **The test:** Start with free. Use it until you hit limits or frustrations. Then decide if paying solves those problems. Don’t pay for features you won’t use. **Q18: Does this AI integrate with tools I already use?** **A:** A tool that works alone is useful. A tool that works with your existing ecosystem is transformative. Integration matters. **Questions to ask:** **Direct integrations:** * Does it connect to tools I use (Slack, Teams, Gmail, Office, etc.)? * Are these integrations native or third-party? * How well do they work? **APIs:** * Does it have an API I can use to build custom integrations? * Is the API well-documented? * What are the API costs? **Export/import:** * Can I easily move data between this tool and others? * Supported formats? * Automation options? **Workflow impact:** * Will this require changing how I work? * Will it save steps or add steps? * Is it additive or disruptive? **Examples:** * A writing AI that integrates with Google Docs saves you copy-paste steps * A design AI that works with Figma or Photoshop fits your workflow * A research AI that connects to your note-taking app becomes part of your system **The risk of standalone:** Tools that don’t integrate create friction. You have to manually move data, switch contexts, remember to use them. Many promising tools die from this friction. **What to do:** Map your current workflow. See where this tool would fit. If integration is missing, decide if the manual steps are worth it—or look for alternatives that integrate better. **Q19: How does this compare to doing it manually?** **A:** This is the ultimate question. AI is a tool, not an end in itself. The comparison should always be: AI vs. current method, not AI vs. nothing. **Compare on multiple dimensions:** **Time:** * How long does it take you to do this manually? * How long with AI (including setup, prompting, verification)? * What’s the net time savings? **Quality:** * Is AI output better, worse, or different? * Does it require significant editing? * For creative work, is the AI version as good as yours? **Cost:** * What’s your time worth? * What does the AI tool cost? * Is the savings worth the expense? **Reliability:** * How consistent is manual work? * How consistent is AI? * What’s the error rate for each? **Learning:** * Do you want to learn/improve this skill yourself? * Is outsourcing to AI depriving you of valuable practice? **Satisfaction:** * Do you enjoy this task? Would you rather keep doing it? * Does using AI make the work more or less fulfilling? **The honest assessment:** Sometimes AI is dramatically better. Sometimes manual is better. Sometimes the answer is “it depends.” Be honest about which applies. **Example:** Writing a thank-you note: manual is faster and more personal. Writing 100 thank-you notes: AI with personalization is vastly better. **The principle:** AI for scale and routine. Manual for quality and personal touch. Choose accordingly. **Q20: What’s the exit strategy if I stop using it?** **A:** The last question is about the future. What happens when you want or need to stop using this tool? **Questions to consider:** **Data export:** Can you get all your data out in usable formats? **Work dependency:** Have you built processes that depend on this tool? How hard would it be to replace? **Skill dependency:** Have you outsourced skills to AI that you’ve lost? Can you function without it? **Contract lock-in:** Are you committed for a minimum term? Auto-renewal you might forget? **Alternatives:** What would you switch to? Have you identified options? **Cost changes:** What if prices triple? Would you still use it? Can you afford to leave? **Company viability:** What if the company goes bankrupt? Shuts down the tool? Gets acquired and changes terms? **The principle:** Always have an exit plan. No tool should become so essential that you can’t function without it. Keep your core skills sharp. Keep your data portable. Keep your options open. **What to do now:** * Export important data periodically * Document your workflows * Maintain manual skills alongside AI use * Keep an eye on alternatives * Read those renewal notices **The bottom line:** Use AI as a tool, not a crutch. Love it while it serves you. Be ready to leave when it doesn’t. *** 💬 Enjoyed this chapter? Have questions or thoughts?\ Join the discussion on GitHub → [**Click here to Comment**](https://github.com/leomohan/theAIhandbook/discussions) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://theaihandbook.leomohan.net/chapter-14-what-questions-should-i-ask-about-ai-tools-before-using-them.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.