DeepSeek‑V3, GPT‑4, Phi‑4, and LLaMA‑3.3: Automating LoRaWAN Engineering with LLM Code Generation
1. Introduction: Engineering Meets Language Models
Low‑Power Wide‑Area Networks (LPWANs), particularly LoRaWAN, are vital technologies for Internet of Things (IoT), enabling long-range, low-data-rate communications in remote areas. Optimizing system parameters—like drone placement and received signal power—requires precise code for path-loss models, geometry, and network dynamics. Traditionally, engineers craft Python functions by hand. But can Large Language Models (LLMs) take over this task?
A recent study evaluated 16 LLMs—including GPT‑4, DeepSeek‑V3, Phi‑4, and LLaMA‑3.3—on progressively complex LoRaWAN tasks described in zero-shot natural language. Results show that both DeepSeek‑V3 and GPT‑4 consistently delivered accurate, executable Python code, while the lightweight Phi‑4 and LLaMA‑3.3 also performed robustly—highlighting that locally run models can be viable alternatives .
2. Problem Setup: From Prompt to Drone Optimization
2.1 Core Tasks
Two tasks were central:
Drone placement optimization: Creating Python routines to determine optimal coordinates given coverage constraints and simulation parameters.
Received power calculation: Calculating path-loss using LoRaWAN’s free-space and empirical models (e.g., Hata model equations).
The tasks ranged in complexity, combining geometry, iterative loops, and domain-specific formulas.
2.2 Evaluation Metric
Each code snippet was extracted, executed, and scored on a 0–5 scale:
0 = error
5 = fully correct logic and output
This grading ensures both functional correctness and performance alignment with real engineering requirements.
3. Models and Setup
3.1 Model Suite
The evaluation included both closed-source (GPT‑4, DeepSeek‑V3) and open‑source models (LLaMA‑3.3, Phi‑4), all accessible locally or via API .
3.2 Prompt Strategy
Prompts were zero-shot and written in plain English, describing tasks like:
“Write a Python function
compute_received_powerthat takes coordinates and returns path-loss power using the free-space model.”
No examples were provided, ensuring the test measured vanilla LM capabilities.
4. Performance Results
4.1 Leading Models: GPT‑4 & DeepSeek‑V3
Both models consistently achieved scores ≥ 4.5/5, handling loops, calculations, and edge cases (e.g., distance zero) reliably.
4.2 Surprising Competitors: Phi‑4 & LLaMA‑3.3
Despite smaller size, these models averaged 3.5–4.0/5, often generating near‑correct functions. Failures typically stemmed from minor syntax issues or missing import statements .
4.3 Weak Models
Other open‑source LLMs scored < 3/5, frequently failing to import math libraries or handling loops incorrectly.
5. Detailed Error Analysis
5.1 Syntax vs. Logic
GPT‑4 and DeepSeek‑V3 rarely failed syntax checks. Phi‑4 and LLaMA‑3.3 errors—often missing colons or indent issues—were easily fixable. Logic errors were rarer than syntax failures.
5.2 Domain Understanding
All models recognized key LoRaWAN equations. Phi‑4 missed a frequency‑based correction factor in ~20% of cases. LLaMA‑3.3 sometimes oversimplified the Hata model.
5.3 Edge Case Handling
GPT‑4 and DeepSeek‑V3 handled edge cases like zero distance, but smaller models often ignored such scenarios, indicating limited robustness.
6. What Makes LLMs Work Here
6.1 Fine-Grained Engineering
DeepSeek‑V3 and GPT‑4 likely benefit from large code‑centric training sets. DeepSeek‑V3's architecture—sparse Mixture‑of‑Experts (MoE) and Multi‑head Latent Attention—optimizes logical reasoning and code coherence .
6.2 Lightweight Viability
Phi‑4, a 14B reasoning‑tuned model, shows that domain‑agnostic fine-tuning can be surprisingly effective .
7. Cost, Inference, and Practicality
7.1 Inference Speed
Locally running models like Phi‑4 and LLaMA‑3.3 enables fast iteration (sub‑second inference). GPT‑4 and DeepSeek‑V3 require API calls or powerful GPUs, impacting latency.
7.2 Model Footprint
LLaMA‑3.3 (~70B) and Phi‑4 (14B) can be hosted on 1–2 consumer GPUs. DeepSeek‑V3 (671B) needs distributed acceleration; GPT‑4 is cloud‑only .
8. Prompt Design Matters
Strict, structured prompts yielded the best results. GPT‑4 and DeepSeek‑V3 were resilient to prompt variance; smaller models required carefully worded instructions.
Even minor tweaks—specifying function signature or error handling—boosted scores by ~5–10%.
9. The Role of Local vs. API Models
The study emphasizes locally deployable models as powerful alternatives to closed APIs. Phi‑4 and LLaMA‑3.3 allow full privacy and adaptability, though fine-tuning or contextual prompt caching may be needed.
10. Domain Fine-Tuning and Future Potential
Injected domain context (e.g., LoRaWAN spec) into prompts or via few-shot examples can elevate smaller models closer to GPT‑4’s performance.
An ideal future pipeline:
Locally run LLaMA‑family model
Domain‑fine-tuned with LoRaWAN snippets
FP‑efficient inference via LoRA adapters
11. Broader Implications
11.1 Engineering Productivity
Automated code scaffolding for numerical calculations and optimization is now practical with LLMs.
11.2 Edge Computing
Local LLMs enable on‑site engineering assistance in field conditions—critical for rural LPWAN deployments.
11.3 Education
Students and technicians can use locally hosted LLMs to prototype solutions without needing cloud access.
12. Limitations & Caveats
Complex tasks like multi-drone coordination or dynamic optimization still require human review.
Robustness can be fragile in edge case handling.
Reliability in production should be ensured via static verification or code testing, not LLM output alone.
13. Recommendation Summary
| Model | Avg Score | Syntax Errors | Domain Errors | Use Case Fit |
|---|---|---|---|---|
| GPT‑4 | 4.8/5 | ≈ 1% | Very rare | Lab-grade reference solution |
| DeepSeek‑V3 | 4.7/5 | ≈ 2% | Very rare | On‑prem/or API engineering support |
| Phi‑4 | 4.0/5 | ≈ 5% | Minor/missing details | Lightweight rapid prototyping |
| LLaMA‑3.3 | 3.8/5 | ≈ 6% | Occasional | Field‑deployable initial draft |
14. Future Work
Multi-turn refinement: Allowing iterative prompt‑feedback cycles
Parameter tuning: Domain-tuning with LoRA or RLHF for LoRaWAN specs
Edge deployment: Test performance on Raspberry Pi–style devices
Tool integration: Plug LLM‑generated code into CI pipelines or static test suites
15. Conclusion
This study underscores that LLMs—large and lean—can reliably generate domain‑specific engineering code. DeepSeek‑V3 and GPT‑4 lead in quality, but Phi‑4 and LLaMA‑3.3 demonstrate local viability. While not yet replacements for seasoned engineers, LLMs are invaluable for bootstrapping code, supporting design, and accelerating computation in IoT engineering.
