Skip to content

Feat/OpenAI agent builtin tools 6657 #6671

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

tejas-dharani
Copy link
Contributor

@tejas-dharani tejas-dharani commented Jun 12, 2025

Why are these changes needed?

This PR adds comprehensive built-in tool support to OpenAIAgent, enabling support for file_search, code_interpreter, web_search_preview, computer_use_preview, image_generation, local_shell and mcp tools. The changes include comprehensive testing for all tool types including mixed usage scenarios, working examples demonstrating tool usage, updated documentation with version notes and usage examples, while maintaining backward compatibility with existing function tools.

Related issue number

Fixes #6657

Checks

…on guide- Fix version format from 0.4.0-dev-1 to 0.4.0-dev.1 for all packages- Remove reference to non-existent Microsoft.AutoGen.Extensions package- Add correct extension packages: Aspire, MEAI, and SemanticKernel- Fix typo: RuntimeGatewway -> RuntimeGateway- Improve documentation structure with clear section headersFixes microsoft#6244
Fix issue microsoft#6277 where TextMessage was used but not imported in three code cells
of the custom agents documentation, causing NameError when users run the examples.

Changes:
- Add TextMessage to imports in ArithmeticAgent section
- Add TextMessage to imports in GeminiAssistantAgent section
- Add TextMessage to imports in Declarative GeminiAssistantAgent section

The CountDownAgent section already had the correct import.

Fixes microsoft#6277
- Support file_search, code_interpreter, web_search_preview, computer_use_preview, image_generation, and mcp tools
- Add comprehensive tests for all tool types including mixed usage scenarios
- Include working examples demonstrating tool usage
- Update documentation with version notes and usage examples
- Maintain backward compatibility with existing function tools

Fixes microsoft#6657
@victordibia
Copy link
Collaborator

@withsmilo
Hi Sungjun,

Would you mid providing a first review here?

@withsmilo
Copy link
Contributor

Sure. Let me review today night.

Copy link

codecov bot commented Jun 14, 2025

Codecov Report

Attention: Patch coverage is 81.92771% with 45 lines in your changes missing coverage. Please review.

Project coverage is 80.40%. Comparing base (9f2c5aa) to head (58ef09c).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...ext/src/autogen_ext/agents/openai/_openai_agent.py 81.92% 45 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6671      +/-   ##
==========================================
+ Coverage   80.35%   80.40%   +0.04%     
==========================================
  Files         233      233              
  Lines       17701    17904     +203     
==========================================
+ Hits        14224    14395     +171     
- Misses       3477     3509      +32     
Flag Coverage Δ
unittests 80.40% <81.92%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…stallation guide- Fix version format from 0.4.0-dev-1 to 0.4.0-dev.1 for all packages- Remove reference to non-existent Microsoft.AutoGen.Extensions package- Add correct extension packages: Aspire, MEAI, and SemanticKernel- Fix typo: RuntimeGatewway -> RuntimeGateway- Improve documentation structure with clear section headersFixes microsoft#6244"

This reverts commit 6d9fb2e.
@tejas-dharani
Copy link
Contributor Author

Thanks for the PR:

  1. Can you add support for built in tools for the JSON config (ie _to_config and _from_config).
  2. Can you add integration tests that hits the actual model api, trigger only when open ai key is in the environment.

Hello Sir,

I have implemented the recommended changes:

  • Added support for built-in tools in the JSON config (_to_config and _from_config).
  • Added integration tests that call the actual model API, which are triggered only when an OpenAI key is present in the environment.

Please let me know if anything else is needed.

@tejas-dharani tejas-dharani requested a review from ekzhu June 28, 2025 07:35
@ekzhu
Copy link
Collaborator

ekzhu commented Jul 6, 2025

@tejas-dharani could you check the tests and doc example errors.

Copy link
Collaborator

@ekzhu ekzhu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please run checks and tests locally first before requesting for review. Thanks.

@tejas-dharani tejas-dharani requested a review from ekzhu July 7, 2025 14:46
@tejas-dharani tejas-dharani requested a review from ekzhu July 10, 2025 15:26
@ekzhu ekzhu enabled auto-merge (squash) July 10, 2025 17:17
@ekzhu ekzhu disabled auto-merge July 10, 2025 17:27
@ekzhu ekzhu merged commit 25d7328 into microsoft:main Jul 10, 2025
69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OpenAIAgent should support other tool types as well as function
5 participants