In order to use ASP.NET AJAX inside an EPiServer custom property you have to have a ScriptManager present. Here's an example of how to add one programmatically.