UE5集成ChatGPT和Stable Diffusion(SD)

前言

最近倒腾了一下之前比较火的AI绘画和ChatGPT,将两者都集成进了UE引擎中进行使用。

具体应用：

• AI绘画，可以用来生成立绘，缓存其生成参数和图片内容。
• ChatGPT，可以用来生成NPC对话。

虽然更多的应用还是离线生成之后，进行配置化。
不过在游戏运行时使用说不定更有趣呢。

---

AI绘画（Stable Diffusion，下简称SD）

1. 环境安装

这里就不过多赘述，网上有很多教程，部署在本地的，远程的都有。

2. 相关Git仓库

• Stable Diffusion 1
  https://github.com/CompVis/stable-diffusion
• Stable Diffusion 2
  https://github.com/Stability-AI/stablediffusion
• Stable Diffusion WebUI (此示例中使用)
  https://github.com/AUTOMATIC1111/stable-diffusion-webui

3.加速(打开xformers)

在命令行设置xformers，加速效果还是很明显的

4.模型

模型去C站上下载：https://civitai.com/

5.命令行参数设置

修改webui-use.bat文件，主要是添加上--api参数，以支持通过HTTP后台API调用

set COMMANDLINE_ARGS= --listen --api --xformers --theme dark --enable-console-prompts

更多命令参数
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Command-Line-Arguments-and-Settings

---

封装SD UE5引擎接口

这里将SD功能引入UE5引擎,本质是SD WebUI作为Web服务器,UE5引擎代替浏览器作为客户端,都是对Http协议的使用和API调用

整体实现思路

• 整理SD WebUI的API的参数
• 通过UE5 HTTP接口请求SD服务
• 处理HTTPResponse，解码图片数据
• 封装为AysncAction，方便蓝图调用，例如应用于UMGImage中

具体步骤

1.SD请求参数

UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
FString Prompt;
UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
FString NegativePrompt;
UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
FString SamplerIndex = TEXT("Euler a");
UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
int Steps = 20;
UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
int Seed = -1;
UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
int Width = 512;
UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
int Height = 512;
UPROPERTY(BlueprintReadWrite, EditAnywhere, meta=(ExposeOnSpawn), Category="BaseParam")
int CfgScale = 7;

2.Http请求接口

void UAsyncTaskRequestSDImage::Start(FString URL, FString JsonParam)
{
	TSharedRef<IHttpRequest, ESPMode::ThreadSafe> HttpRequest = FHttpModule::Get().CreateRequest();
	HttpRequest->SetURL(URL);
	// FString JsonStr = "{\"prompt\": \"a beautiful girl\",\"negative_prompt\": \"\",\"sampler_index\": \"Euler a\",\"seed\": 3,\"steps\": 20,\"width\": 512,\"height\": 512,\"cfg_scale\": 7}";
	HttpRequest->SetHeader(TEXT("Content-Type"), TEXT("application/json; charset=utf-8"));
	HttpRequest->SetVerb(TEXT("POST"));
	HttpRequest->SetContentAsString(JsonParam);
	HttpRequest->OnProcessRequestComplete().BindUObject(this, &UAsyncTaskRequestSDImage::HandleImageRequest);
	HttpRequest->ProcessRequest();
}

3.处理HTTPResponse，解码图片数据

void UAsyncTaskRequestSDImage::HandleImageRequest(FHttpRequestPtr HttpRequest, FHttpResponsePtr HttpResponse,
                                                  bool bSucceeded)
{
	if (bSucceeded && HttpResponse.IsValid() && HttpResponse->GetResponseCode() == 200 && HttpResponse->
		GetContentLength() > 0)
	{
		FString JsonStr = HttpResponse->GetContentAsString();
		TSharedRef<TJsonReader<TCHAR>> JsonReader = TJsonReaderFactory<TCHAR>::Create(JsonStr);
		TSharedPtr<FJsonValue> CurrentJsonValue;
		FJsonSerializer::Deserialize(JsonReader, CurrentJsonValue);
		TArray<TSharedPtr<FJsonValue>> ImageJsonObjectArray = CurrentJsonValue->AsObject()->GetArrayField(
			TEXT("images"));

		IImageWrapperModule& ImageWrapperModule = FModuleManager::LoadModuleChecked<IImageWrapperModule>(
			FName("ImageWrapper"));
		TSharedPtr<IImageWrapper> ImageWrappers[3] =
		{
			ImageWrapperModule.CreateImageWrapper(EImageFormat::PNG),
			ImageWrapperModule.CreateImageWrapper(EImageFormat::JPEG),
			ImageWrapperModule.CreateImageWrapper(EImageFormat::BMP),
		};

		for (int j = 0; j < ImageJsonObjectArray.Num(); j++)
		{
			auto CurrentImage = ImageJsonObjectArray[j]->AsString();
			TArray<uint8> CurrentImageBytes;
			FBase64::Decode(CurrentImage, CurrentImageBytes);
			for (auto ImageWrapper : ImageWrappers)
			{
				if (ImageWrapper.IsValid() && ImageWrapper->SetCompressed(
					CurrentImageBytes.GetData(), CurrentImageBytes.GetAllocatedSize()))
				{
					TArray64<uint8> RawData;
					const ERGBFormat InFormat = ERGBFormat::BGRA;
					if (ImageWrapper->GetRaw(InFormat, 8, RawData))
					{
						if (UTexture2DDynamic* Texture = UTexture2DDynamic::Create(
							ImageWrapper->GetWidth(), ImageWrapper->GetHeight()))
						{
							Texture->SRGB = true;
							Texture->UpdateResource();

							FTexture2DDynamicResource* TextureResource = static_cast<FTexture2DDynamicResource*>(Texture
								->GetResource());
							if (TextureResource)
							{
								ENQUEUE_RENDER_COMMAND(FWriteRawDataToTexture)(
									[TextureResource, RawData = MoveTemp(RawData)](FRHICommandListImmediate& RHICmdList)
									{
										TextureResource->WriteRawToTexture_RenderThread(RawData);
									});
							}
							OnSuccess.Broadcast(Texture);
							return;
						}
					}
				}
			}
		}
	}
	OnFail.Broadcast(nullptr);
}

4.封装为AysncAction，方便蓝图调用(见源码)

继承AysncActionBase

使用示例

• 先启动SD WebUI，打开其API调用功能
  
• 示例蓝图BP_SDSample
  构造参数，发起请求，回调设置Image
  
• 生成结果
  

---

封装GPT UE5引擎接口

整体实现思路

• 整理GPT参数
• 通过UE5 HTTP接口请求GPT服务
• 处理HTTPResponse和Progress
• 封装为AysncAction，方便蓝图调用

具体步骤

1.整理GPT参数

FString Content;  //问题内容
FString Key;  //api-key

2.通过UE5 HTTP接口请求GPT服务

void UAsyncTaskRequestChatGPT::Start(FString Content, FString Key)
{
	TSharedPtr<FJsonObject> HeaderObjectJson = MakeShareable(new FJsonObject);
	TSharedPtr<FJsonObject> ModelObjectJson = MakeShareable(new FJsonObject);
	ModelObjectJson->SetStringField(TEXT("id"), TEXT("gpt-3.5-turbo"));
	ModelObjectJson->SetStringField(TEXT("name"), TEXT("GPT-3.5"));
	ModelObjectJson->SetNumberField(TEXT("maxLength"), 12000);
	ModelObjectJson->SetNumberField(TEXT("tokenLimit"), 4000);
	HeaderObjectJson->SetObjectField(TEXT("model"), ModelObjectJson);

	TArray<TSharedPtr<FJsonValue>> JsonArray;

	TSharedPtr<FJsonObject> OneTempJsonObject = MakeShareable(new FJsonObject);
	OneTempJsonObject->SetStringField(TEXT("role"), TEXT("user"));
	OneTempJsonObject->SetStringField(TEXT("content"), Content);
	TSharedPtr<FJsonValue> FirstJsonValue = MakeShareable(new FJsonValueObject(OneTempJsonObject));
	JsonArray.Add(FirstJsonValue);
	HeaderObjectJson->SetArrayField(TEXT("messages"), JsonArray);

	HeaderObjectJson->SetStringField(TEXT("key"), Key);
	HeaderObjectJson->SetStringField(
		TEXT("prompt"),
		TEXT(
			"You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown."));
	HeaderObjectJson->SetNumberField(TEXT("temperature"), 1);

	FString HeaderString;
	TSharedRef<TJsonWriter<>> Writer = TJsonWriterFactory<>::Create(&HeaderString);
	FJsonSerializer::Serialize(HeaderObjectJson.ToSharedRef(), Writer);
	HeaderString = HeaderString.Replace(TEXT("\n"), TEXT(""));
	HeaderString = HeaderString.Replace(TEXT("\r"), TEXT(""));
	HeaderString = HeaderString.Replace(TEXT("\t"), TEXT(""));

	TSharedRef<IHttpRequest, ESPMode::ThreadSafe> HttpRequest = FHttpModule::Get().CreateRequest();
	FString URL;
	check(GWorld);
	USDIntegrationSubsystem* SDIntegrationSubsystem = GWorld->GetGameInstance()->GetSubsystem<
		USDIntegrationSubsystem>();
	check(SDIntegrationSubsystem);
	SDIntegrationSubsystem->GetRequestChatGPTURL(URL);
	HttpRequest->SetURL(URL);
	HttpRequest->SetHeader(TEXT("Content-Type"), TEXT("application/json; charset=utf-8"));
	HttpRequest->SetHeader(
		TEXT("User-Agent"),
		TEXT(
			"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"));
	HttpRequest->SetVerb(TEXT("POST"));
	
	HttpRequest->SetContentAsString(HeaderString);
	HttpRequest->OnRequestProgress().BindUObject(this, &UAsyncTaskRequestChatGPT::HandleProgress);
	HttpRequest->OnProcessRequestComplete().BindUObject(this, &UAsyncTaskRequestChatGPT::HandleChatGPTRequest);
	HttpRequest->ProcessRequest();
}

3.处理HTTPResponse和Progress

处理Progress,可以在回调中进行文本的设置，可以实现类似打印机的效果

void UAsyncTaskRequestChatGPT::HandleProgress(FHttpRequestPtr Request, int32 BytesSent, int32 BytesReceived)
{
	UE_LOG(LogTemp, Log, TEXT("HandleProgress:%s"), *Request->GetResponse()->GetContentAsString());
	OnProgress.Broadcast(Request->GetResponse()->GetContentAsString());
}

4.封装为AysncAction，方便蓝图调用(见源码)

继承AysncActionBase

使用示例

• 在初始化中设置一个全局的Api-Key
  

• 蓝图BP_ChatGPTSample,调用异步任务节点即可
  

• 生成结果
  

UE5 SDIntegration Plugin v1

https://github.com/CalmLoader/ChatGPT-SD-Integration-UE5

---

其他问题

Lora模型没法加载的问题
https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/7984

PostCat（API测试工具）
https://github.com/Postcatlab/postcat

参考链接

https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/API
https://zhuanlan.zhihu.com/p/629845245
https://zhuanlan.zhihu.com/p/624042359