Role overview
Develop local LLM applications and agent systems for AI PCs and smart computing devices with new inference chips. Integrate model deployment, inference runtimes, application development and performance optimization with the device’s compute, memory and inference architecture to turn hardware capabilities into deliverable product experiences.
Responsibilities
- Develop local AI applications and agents for knowledge assistance, content processing, developer support and office automation, from requirements analysis and architecture through implementation and delivery.
- Integrate chip SDKs and inference runtimes: connect vendor model interfaces, runtimes and toolchains; implement model loading, inference calls, streaming output, cancellation and resource cleanup behind unified application interfaces.
- Adapt and deploy models based on hardware support, including model selection, format conversion, quantization validation and compatibility testing; evaluate quality, context length and resource consumption.
- Build agent execution capabilities including tool calling, task orchestration, context and memory management, persistent state and error recovery; connect local files, knowledge bases, applications and system tools.
- Optimize system-level application performance by identifying bottlenecks in model loading, prefill, decode, retrieval and tool execution. Improve time to first token, generation speed, peak memory and task completion time while accounting for power use and sustained operation.
- Adapt to heterogeneous computing environments: design task allocation, model execution and fallback strategies for CPUs, GPUs, NPUs or dedicated inference accelerators.
- Build evaluation and debugging systems with test sets covering model quality, agent task success, device performance and stability; support comparisons and regression testing across models, chips and runtime versions.
- Work with chip, compiler, inference engine and product teams to diagnose software/hardware integration issues and validate new inference architectures in real applications.
Requirements
- Master’s degree in computing, software engineering, electronic engineering or a related field; at least three years of software development experience, including local AI applications, inference deployment or software/hardware integration projects.
- Proficiency in at least one of Python, C++ or Rust; able to read SDK documentation and examples, call native libraries, wrap interfaces and debug integrations.
- Familiarity with development on Windows or Linux, including multithreading, asynchronous tasks, interprocess communication, memory management and performance analysis; able to independently diagnose application stability issues.
- Understand LLM inference, including model weights, tokenizers, prefill/decode, KV cache, context length and quantization precision, and analyze their impact on device resources and application experience.
- Hands-on experience with at least one local inference framework or chip toolchain, such as llama.cpp, ONNX Runtime, OpenVINO, TensorRT or a vendor inference SDK.
- Able to deploy models and validate performance on target devices, with an understanding of operator support, model conversion, dynamic shapes, data movement and hardware fallback.
- LLM application or agent development experience; familiarity with prompts, structured output, tool calling, RAG and context management, and the ability to address local models’ limitations in instruction following and tool use.
- Strong experimentation and evaluation skills: quantify trade-offs among quality, latency, throughput and memory with clearly specified hardware, model versions and test conditions.
- Interest in new AI inference architectures; able to troubleshoot independently amid evolving toolchains and incomplete documentation, and collaborate effectively with low-level engineering teams.
Nice to have
- Experience delivering AI PCs, workstations, local AI appliances or dedicated inference devices;
- adapting applications to new NPUs, dataflow architectures, compute-in-memory or other dedicated inference architectures;
- practical optimization involving heterogeneous computing, memory bandwidth constraints, zero-copy or layer-wise model execution;
- model quantization and accuracy regression testing, mixed precision, long-context optimization or speculative decoding;
- compiler, operator or inference engine development and low-level performance or compatibility troubleshooting;
- local multimodal, voice interaction, document understanding, coding agent or desktop automation projects;
- offline deployment, model distribution and updates, device compatibility testing or observability.
Application materials
Please include a project demo or code repository and describe the target device and chip, model and quantization specifications, inference runtime, your contribution, and measured performance and task outcomes.