Testing MCP Autostart Behavior And Trust In VS Code

Hey everyone! In this article, we're diving deep into the new autostart behavior for MCP (Microsoft Chat Platform) servers in VS Code. This is a crucial update that streamlines the chat experience, and we need your help to ensure everything runs smoothly. We'll walk you through the new features, how to test them, and what to look for. So, let's get started!

Understanding the New Autostart Feature

In this latest iteration, a new setting, chat.mcp.autostart, has been introduced, which is currently enabled by default. This feature is designed to automatically start MCP servers, ensuring they're ready to go when you need them. The main goal here is to make sure each MCP server is started at least once, so VS Code knows exactly what tools it provides. These tool caches can be easily cleared using the MCP: Reset Tool Caches command. This is important for maintaining a clean and efficient chat environment.

With the chat.mcp.autostart setting turned on, you should notice a significant change in behavior. Previously, you might have seen a blue refresh icon indicating that a server needed to be started manually. Now, the servers should automatically start when you submit a chat request. This means less waiting and a more seamless experience. To verify that the servers are indeed starting automatically, you can check their status in the MCP: List Servers view. This will give you a clear picture of which servers are running and ready to assist.

Testing the Autostart Feature

To thoroughly test this new feature, you'll need to install some MCP servers in both your user settings and workspace settings. This will allow you to simulate different scenarios and ensure the autostart functionality works consistently across various configurations. You can add these servers by modifying your user settings and workspace settings in VS Code.

Key Test Cases

1. Automatic Server Start: With the chat.mcp.autostart setting enabled, verify that you no longer see the blue refresh icon. Instead, the servers should start automatically when you submit a chat request. You can confirm this by checking the server status in MCP: List Servers.
2. Tool Access: Ensure that the chat has access to the tools provided by the newly started MCP servers in the first chat request itself. There are two ways to verify this:
   • Ask a question that would naturally lead the model to use a specific tool. If the model responds appropriately, it indicates that the tool is accessible.
   • Check the conversation log to see if the tool is listed as preset. This provides a direct confirmation that the server's tools are available.
3. Workspace Trust Prompts: Servers installed in the workspace should trigger a trust prompt initially when started, and also whenever their configuration is changed. This is a crucial security measure to ensure that you are aware of and approve the servers being used in your workspace. Specifically, try updating the configuration of a workspace server to see if it triggers the trust prompt. The trust prompt should appear when the server is auto-started after the configuration change. Take your time to carefully review the dialog options and make sure they make sense in the context of your workflow. The dialog should also handle scenarios where multiple servers are prompting for trust simultaneously. If you choose to cancel out of the trust flow, the chat request should proceed without those specific servers.
4. Denied Trust Behavior: If trust is denied to one or more servers, they should not prompt for trust again automatically. This prevents a frustrating cycle of repeated prompts. However, you should still be able to manually start these servers. You can do this either through the MCP: List Servers view or via the code lens in the mcp.json file. It's important to note that we don't trigger a trust prompt for the code lens action, as the user is already actively reviewing their configuration when using this feature. The trust state is remembered based on the server name, so you can easily manage which servers are trusted.
5. chat.mcp.autostart set to never: If the chat.mcp.autostart setting is explicitly set to never, the servers should not autostart at all. In this case, you should observe the old behavior, where servers need to be started manually. This allows users who prefer the manual control to maintain their workflow.

Understanding Trust Prompts and Their Importance

The trust prompts are a critical part of the MCP server security model. They ensure that you, the user, are always in control of which servers are running and have access to your data. When a workspace server's configuration is changed, the trust prompt is re-triggered to make sure you're aware of any updates and can re-evaluate your trust decision. This is especially important in collaborative environments where multiple people might be contributing to a workspace.

The trust dialog should handle multiple servers prompting for trust in the same dialog. This is a common scenario, especially when you're adding or updating several servers at once. The dialog should clearly list each server and allow you to make an informed decision about whether to trust it. If you cancel out of the trust flow, the chat request should proceed without those servers, ensuring that you're not blocked from using the chat functionality while you're making trust decisions. This provides a balance between security and usability.

Handling Denied Trust and Manual Server Management

If you deny trust to one or more servers, they should not prompt you again automatically. This prevents a disruptive user experience. However, there should still be a way to manually start these servers if you change your mind or need them for a specific task. This is where the MCP: List Servers view and the code lens in the mcp.json file come into play. These tools provide you with the flexibility to manage your MCP servers on your terms.

The code lens integration is particularly useful because it allows you to start a server directly from its configuration file. This is convenient when you're actively working on the configuration and need to test the server. We intentionally don't trigger a trust prompt for the code lens action because you're already looking at the server's configuration and are likely to be making an informed decision. Trust state is remembered based on the server name, so you don't have to worry about re-prompting for the same server repeatedly.

Testing the chat.mcp.autostart Setting Set to never

For users who prefer the manual control of starting MCP servers, the chat.mcp.autostart setting can be set to never. When this setting is enabled, the servers should not autostart, and the user should experience the previous behavior. This ensures that those who prefer the manual approach can continue to work in a way that suits them best. To verify this, simply set the chat.mcp.autostart setting to never in your VS Code settings and observe that the servers do not start automatically when you submit a chat request.

Conclusion: Your Feedback Matters!

This new autostart feature for MCP servers is a significant step towards improving the chat experience in VS Code. By automatically starting the servers, we aim to reduce friction and make it easier for you to access the tools you need. However, we need your help to ensure that this feature works flawlessly. Please take the time to test the scenarios outlined in this article and provide feedback on your experience.

Your feedback is invaluable in helping us identify and fix any issues. By working together, we can make VS Code an even better tool for developers. So, go ahead, install some MCP servers, play around with the settings, and let us know what you think. Happy testing, folks! And thanks for being awesome and helping us make VS Code the best it can be. Remember, your insights and experiences are what drive us to improve and innovate. So, don't hesitate to share your thoughts and help us make this feature rock!