Microphone Text Input Mod User Documention

Architectury 2.x

Introduction

The Microphone Text Input Mod is a Fabric mod designed for Minecraft clients. It provides speech recognition input functionality and automatically converts spoken words into text chat messages, enhancing the in - game communication experience.

Features

• Multiple Modes: Supports different operation modes, including AUTO_SEND, RELEASE_KEY_TO_SEND, and RELEASE_KEY_TO_INPUT.
• Advanced Configuration: Allows users to adjust advanced parameters related to the Whisper library, such as the number of threads, audio context size, and sampling strategy.

Dependencies

Dependency Name Fabric NeoForge Java 21 21 Fabric API see fabric.mod.json ❌ Architectury API ❌ see neoforge.mods.toml MidnightLib see fabric.mod.json see neoforge.mods.toml

Configuration

Keybinding

Keybinding Name Recognize Category key.category.minecraft.mcmti Translation Key key.mcmti.recognize Default Key V

General Settings

Setting Translation Key Field Type Default Value Description GGML Whisper Model mcmti.midnightconfig.model me.jaffe2718.mcmti.config.McmtiConfig.model String "" Path to the GGML Whisper model. Language mcmti.midnightconfig.language me.jaffe2718.mcmti.config.McmtiConfig.language String "en" Language for speech recognition. Mode mcmti.midnightconfig.mode me.jaffe2718.mcmti.config.McmtiConfig.mode me.jaffe2718.mcmti.config.McmtiConfig.Mode "RELEASE_KEY_TO_SEND" Mod's work mode. Record Cycle (ms) mcmti.midnightconfig.recordCycleMs me.jaffe2718.mcmti.config.McmtiConfig.recordCycleMs int 5000 Record cycle in milliseconds. Record Buffer Size (byte) mcmti.midnightconfig.recordBufferSize me.jaffe2718.mcmti.config.McmtiConfig.recordBufferSize int 1024 Record buffer size in bytes. Prefix mcmti.midnightconfig.prefix me.jaffe2718.mcmti.config.McmtiConfig.prefix String "⌈Speech Input⌋" Prefix added to the recognized text. Draft Input mcmti.midnightconfig.draftInput me.jaffe2718.mcmti.config.McmtiConfig.draftInput boolean false Enable draft input. If enabled, the recognized text will be shown in the entry as a draft. Encoding Repair mcmti.midnightconfig.encodingRepair me.jaffe2718.mcmti.config.McmtiConfig.encodingRepair boolean false Enable encoding repair. Source Encoding mcmti.midnightconfig.srcEncoding me.jaffe2718.mcmti.config.McmtiConfig.srcEncoding String Charset.defaultCharset().displayName() Source encoding for text. Applies only if encoding repair is enabled. Destination Encoding mcmti.midnightconfig.dstEncoding me.jaffe2718.mcmti.config.McmtiConfig.dstEncoding String Charset.defaultCharset().displayName() Destination encoding for text. Applies only if encoding repair is enabled.

Advanced Settings

Setting Translation Key Field Type Default Value Description Enable Advanced Config mcmti.midnightconfig.advancedConfig me.jaffe2718.mcmti.config.McmtiConfig.advancedConfig boolean false Enable advanced configuration. nThreads mcmti.midnightconfig.nThreads me.jaffe2718.mcmti.config.McmtiConfig.nThreads int 0 Number of threads to use for the operation of the Whisper model. 0 for max cores. audioCtx mcmti.midnightconfig.audioCtx me.jaffe2718.mcmti.config.McmtiConfig.audioCtx int 0 Audio context size for the Whisper model. 0 means use default. nMaxTextCtx mcmti.midnightconfig.nMaxTextCtx me.jaffe2718.mcmti.config.McmtiConfig.nMaxTextCtx int 16384 Max tokens to use from past text as prompt for the decoder. offsetMs mcmti.midnightconfig.offsetMs me.jaffe2718.mcmti.config.McmtiConfig.offsetMs int 0 Offset in ms to start recording from. durationMs mcmti.midnightconfig.durationMs me.jaffe2718.mcmti.config.McmtiConfig.durationMs int 0 Audio duration to process in ms. 0 means use default. translate mcmti.midnightconfig.translate me.jaffe2718.mcmti.config.McmtiConfig.translate boolean false Translate the text to the default language. noTimestamps mcmti.midnightconfig.noTimestamps me.jaffe2718.mcmti.config.McmtiConfig.noTimestamps boolean false Do not generate timestamps. initialPrompt mcmti.midnightconfig.initialPrompt me.jaffe2718.mcmti.config.McmtiConfig.initialPrompt String "" Initial text to use as a prompt for the whisper. noContext mcmti.midnightconfig.noContext me.jaffe2718.mcmti.config.McmtiConfig.noContext boolean true Do not use past transcription (if any) as initial prompt for the decoder. singleSegment mcmti.midnightconfig.singleSegment me.jaffe2718.mcmti.config.McmtiConfig.singleSegment boolean false Force single segment output (useful for streaming). printSpecial mcmti.midnightconfig.printSpecial me.jaffe2718.mcmti.config.McmtiConfig.printSpecial boolean false Print special tokens. printProgress mcmti.midnightconfig.printProgress me.jaffe2718.mcmti.config.McmtiConfig.printProgress boolean true Print progress information. printRealtime mcmti.midnightconfig.printRealtime me.jaffe2718.mcmti.config.McmtiConfig.printRealtime boolean false Print results from within whisper.cpp (avoid it, use callback instead). printTimestamps mcmti.midnightconfig.printTimestamps me.jaffe2718.mcmti.config.McmtiConfig.printTimestamps boolean true Print timestamps for each text segment when printing realtime. suppressBlank mcmti.midnightconfig.suppressBlank me.jaffe2718.mcmti.config.McmtiConfig.suppressBlank boolean true Decoder option. suppressNonSpeechTokens mcmti.midnightconfig.suppressNonSpeechTokens me.jaffe2718.mcmti.config.McmtiConfig.suppressNonSpeechTokens boolean true Tokenizer option. temperature mcmti.midnightconfig.temperature me.jaffe2718.mcmti.config.McmtiConfig.temperature float 0.0f Initial decoding temperature. maxInitialTs mcmti.midnightconfig.maxInitialTs me.jaffe2718.mcmti.config.McmtiConfig.maxInitialTs float 1.0f Maximum initial timestamp. lengthPenalty mcmti.midnightconfig.lengthPenalty me.jaffe2718.mcmti.config.McmtiConfig.lengthPenalty float -1.0f Length penalty. temperatureInc mcmti.midnightconfig.temperatureInc me.jaffe2718.mcmti.config.McmtiConfig.temperatureInc float 0.4f Temperature increment. entropyThold mcmti.midnightconfig.entropyThold me.jaffe2718.mcmti.config.McmtiConfig.entropyThold float 2.4f Entropy threshold (similar to OpenAI's "compression_ratio_threshold"). logprobThold mcmti.midnightconfig.logprobThold me.jaffe2718.mcmti.config.McmtiConfig.logprobThold float -1.0f Log probability threshold. noSpeechThold mcmti.midnightconfig.noSpeechThold me.jaffe2718.mcmti.config.McmtiConfig.noSpeechThold float 0.6f No speech threshold. greedyBestOf mcmti.midnightconfig.greedyBestOf me.jaffe2718.mcmti.config.McmtiConfig.greedyBestOf int -1 Specific to greedy sampling strategy. beamSearchBeamSize mcmti.midnightconfig.beamSearchBeamSize me.jaffe2718.mcmti.config.McmtiConfig.beamSearchBeamSize int 2 Specific to bean search sampling strategy. beamSearchPatience mcmti.midnightconfig.beamSearchPatience me.jaffe2718.mcmti.config.McmtiConfig.beamSearchPatience float -1.0f Specific to bean search sampling strategy. grammar mcmti.midnightconfig.grammar me.jaffe2718.mcmti.config.McmtiConfig.grammar String "" Grammar file path. Empty means no grammar. grammarPenalty mcmti.midnightconfig.grammarPenalty me.jaffe2718.mcmti.config.McmtiConfig.grammarPenalty float 100.0f Penalty for non grammar tokens. whisperSamplingStrategy mcmti.midnightconfig.whisperSamplingStrategy me.jaffe2718.mcmti.config.McmtiConfig.whisperSamplingStrategy io.github.freshsupasulley.whisperjni.WhisperSamplingStrategy BEAM_SEARCH The WhisperContext enum to configure whisper's sampling strategy. vad mcmti.midnightconfig.vad me.jaffe2718.mcmti.config.McmtiConfig.vad boolean false Enable VAD (Voice Activity Detection). vad__max_speech_duration_s mcmti.midnightconfig.vad__max_speech_duration_s me.jaffe2718.mcmti.config.McmtiConfig.vad__max_speech_duration_s float 0f Max duration of a speech segment before forcing a new segment. vad__min_silence_duration_ms mcmti.midnightconfig.vad__min_silence_duration_ms me.jaffe2718.mcmti.config.McmtiConfig.vad__min_silence_duration_ms int 0 Min silence duration to consider speech as ended. vad__min_speech_duration_ms mcmti.midnightconfig.vad__min_speech_duration_ms me.jaffe2718.mcmti.config.McmtiConfig.vad__min_speech_duration_ms int 0 Min duration for a valid speech segment. vad__samples_overlap mcmti.midnightconfig.vad__samples_overlap me.jaffe2718.mcmti.config.McmtiConfig.vad__samples_overlap float 0f Overlap in seconds when copying audio samples from speech segment. vad__speech_pad_ms mcmti.midnightconfig.vad__speech_pad_ms me.jaffe2718.mcmti.config.McmtiConfig.vad__speech_pad_ms int 0 Padding added before and after speech segments. vad__threshold mcmti.midnightconfig.vad__threshold me.jaffe2718.mcmti.config.McmtiConfig.vad__threshold float 0f Probability threshold to consider as speech. vad_model_path mcmti.midnightconfig.vad_model_path me.jaffe2718.mcmti.config.McmtiConfig.vad_model_path String "" Path to the VAD model. Empty means use default. useCustomDynamicLib mcmti.midnightconfig.useCustomDynamicLib me.jaffe2718.mcmti.config.McmtiConfig.useCustomDynamicLib boolean false Enable using custom dynamic library for Whisper. customDynamicLibDir mcmti.midnightconfig.customDynamicLibDir me.jaffe2718.mcmti.config.McmtiConfig.customDynamicLibDir String "" Custom dynamic link library directory for Whisper.

• For more info, see WhisperFullParams.java

WARNING: Activating the advanced configuration will change the default parameters of the Whisper model configuration, which will have a critical impact on the speech recognition results. Inappropriate configuration of advanced parameters can lead to problems such as speech recognition failure, high computer resource usage, and program crashes. Please use with caution.

Usage

1. Install the mod and download the GGML Whisper model.
2. Configure the mod according to your needs in the configuration menu.
3. Use the keybinding you set (default is V) to start recording and speech recognition.
4. If the mode is set to AUTO_SEND, the recognized text will be automatically sent as a chat message. If set to RELEASE_KEY_TO_SEND or RELEASE_KEY_TO_INPUT, follow the corresponding key - release actions.

Use VAD

If you want to use VAD (Voice Activity Detection), you need to download VAD model, enable the advanced configuration and set the vad_model_path to the path of the VAD model in the configuration menu.

Custom Dynamic Library

1. Download the custom dynamic library from Jaffe2718/whisper-jni and extract the files.
2. Enable the advanced configuration and set the useCustomDynamicLib to true in the configuration menu.
3. Set the customDynamicLibDir to the directory where the custom dynamic library is located in the configuration menu.
4. If you want to use the dynamic library which is supported vulkan, check your check that your computer has drivers and libraries running Vulkan installed. vulkaninfo
5. If you want to use CUDA, check that your computer has drivers and libraries running CUDA installed. nvidia-smi For Linux, you need to install CUDA Toolkit >= 12.4.0 and configure the environment variables. For Windows, if the game crashes, you have to force the game to use the Java >= 25, see Jaffe2718/whisper-jni/v0.5.6

Troubleshooting

• Crash on Startup: Please check the compatibility and version of your use of the dynamic link library. You can disable the advanced configuration and set the useCustomDynamicLib to false to use the default dynamic library (CPU version).
• Audio Input Device Load Failed: Please check if Java has access to the audio input device.
• Whisper Model Load Failed: Make sure the path to the GGML Whisper model is correct.

Contributing

If you'd like to contribute to this project, please feel free to submit issues or pull requests on GitHub.