Evaluating LLMs for LoRaWAN Engineering Tasks: DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3 in Code Generation

Introduction

The rapid progress of large language models (LLMs) has brought forward a new era in computational problem solving, where natural language can serve as the interface to complex engineering tasks. While the majority of LLM research has centered on general-purpose reasoning, creative writing, or open-domain coding assistance, there is increasing demand for assessing whether these models can be applied to highly specialized technical areas. One such area is LoRaWAN (Long Range Wide Area Network), a low-power, wide-area networking protocol that supports applications such as IoT sensor deployment, drone-assisted data collection, and remote communications.

This article focuses on how LLMs handle LoRaWAN-related engineering tasks, particularly when asked to generate correct Python code from progressively complex zero-shot natural language prompts. We examine whether lightweight LLMs running locally—such as Phi-4 and LLaMA-3.3—can realistically compete with state-of-the-art, resource-intensive systems like DeepSeek-V3 and OpenAI GPT-4.

The research is grounded in two core engineering challenges:

1. Optimal drone positioning for LoRaWAN coverage.

2. Received power calculations based on propagation models.

We benchmarked 16 models in total, but this article emphasizes comparative findings across four key representatives: DeepSeek-V3, GPT-4, Phi-4, and LLaMA-3.3. By analyzing code accuracy, execution reliability, and error types, we aim to highlight both the potential and limitations of using LLMs in specialized engineering domains.

Background: LoRaWAN Engineering Challenges

LoRaWAN is a critical enabler of IoT connectivity, supporting applications where long-range communication must occur under strict power and bandwidth constraints. Two problems often faced by engineers are:

1. Drone-Assisted LoRaWAN Coverage
   Drones can act as mobile gateways or relays for IoT sensors in areas with limited infrastructure. Optimizing drone positioning ensures maximum coverage while minimizing energy use. This involves computational geometry, optimization algorithms, and physical constraints such as altitude, antenna orientation, and path loss.

2. Received Power Estimation
   Accurate link-budget analysis requires applying propagation models (e.g., Free Space Path Loss, Okumura-Hata, or log-distance models). Engineers typically rely on Python or MATLAB implementations to calculate received power under different conditions.

Both tasks demand not only domain-specific knowledge but also the ability to translate abstract concepts into executable code. LLMs, if capable, could accelerate prototyping and reduce the barrier to entry for engineers.

Methodology

1. Model Selection

We tested 16 models, but the article narrows focus to four for clarity:

• DeepSeek-V3: A cutting-edge model known for high performance in coding tasks, with strong mathematical reasoning abilities.

• GPT-4: OpenAI’s flagship model, widely regarded as the benchmark in accuracy and reliability.

• Phi-4: A lightweight Microsoft model optimized for efficient local execution.

• LLaMA-3.3: A Meta-developed model, smaller in scale but praised for balance between reasoning quality and efficiency.

2. Prompting Strategy

Tasks were framed using zero-shot natural language prompts. For example:

• “Write a Python function that calculates the received power of a LoRaWAN signal given frequency, distance, and antenna gains using the free-space path loss model.”

• “Write a Python function that determines the optimal drone altitude to maximize LoRaWAN coverage radius while minimizing power consumption.”

Prompts were progressively made more complex, testing model robustness.

3. Evaluation Framework

Each generated Python function was:

1. Extracted and executed in a controlled runtime.

2. Scored on a 0–5 scale, where:

• 0 = Non-compilable code.

• 1 = Code compiles but produces irrelevant results.

• 2 = Minor errors requiring debugging.

• 3 = Functional but incomplete or inaccurate.

• 4 = Correct with minor limitations.

• 5 = Fully correct and executable as intended.

We supplemented scoring with qualitative observations, including error types and prompt sensitivity.

Results

1. GPT-4 and DeepSeek-V3: Consistent High Accuracy

• GPT-4 demonstrated exceptional reliability, consistently generating correct code with minimal need for debugging. For both drone positioning and received power calculation, GPT-4 produced Python functions rated 5/5 in over 90% of trials.

• DeepSeek-V3 rivaled GPT-4, occasionally outperforming in efficiency. For example, in received power calculations, DeepSeek-V3 often wrote concise, vectorized code leveraging NumPy, while GPT-4 tended toward more verbose implementations.

2. Phi-4: Small but Surprisingly Capable

Despite being a lightweight model, Phi-4 frequently achieved scores of 3 or 4, producing runnable code with minor syntax or logic errors. Notably, Phi-4 excelled in path loss computations, producing formulas almost identical to textbook references. However, it struggled with more abstract tasks like optimization, sometimes defaulting to brute-force approaches instead of efficient algorithms.

3. LLaMA-3.3: Mixed Performance

LLaMA-3.3 showed moments of strength but lacked consistency. In several cases, it misinterpreted the task, confusing LoRaWAN with general wireless communication protocols. Its code often required manual debugging due to missing imports, incorrect function structures, or incomplete logic. Scores ranged widely from 1 to 4.

4. Error Analysis

Common failure modes across all models included:

• Misinterpreting domain-specific terms (e.g., treating “drone coverage” as camera coverage instead of communication coverage).

• Syntax errors in smaller models due to incomplete code generation.

• Overgeneralization, where models substituted generic wireless formulas instead of LoRaWAN-specific parameters.

• Numerical inconsistencies, especially in models lacking strong mathematical grounding.

Discussion

1. Feasibility of Lightweight Models

The performance of Phi-4 and LLaMA-3.3 suggests that lightweight models can indeed contribute meaningfully to engineering workflows, particularly when local execution is critical for privacy or resource constraints. While they fall short of GPT-4 and DeepSeek-V3 in reliability, their outputs are often “good enough” with minimal debugging by a domain expert.

2. Role of Prompt Engineering

We observed significant sensitivity to prompt phrasing. For example, when explicitly asked to use “free space path loss (FSPL) at 868 MHz,” smaller models performed better, while vague prompts like “calculate received power” led to generic or erroneous outputs. This reinforces the necessity of rigorous prompt design for specialized applications.

3. Implications for Engineering Workflows

The findings suggest a hybrid approach:

• Heavyweight LLMs (DeepSeek-V3, GPT-4) for mission-critical tasks requiring high accuracy.

• Lightweight LLMs (Phi-4, LLaMA-3.3) as offline assistants for quick prototyping, educational purposes, or field scenarios with limited connectivity.

4. Future Opportunities

Further research could explore:

• Domain-specific fine-tuning of lightweight models on LoRaWAN datasets.

• Integration of symbolic solvers with LLM reasoning to reduce numerical errors.

• Benchmarks combining LLM outputs with traditional simulation tools such as ns-3 or MATLAB.

Case Study: Drone Optimal Positioning

To illustrate, we compare outputs from each model on the prompt:

“Write a Python function to compute the optimal altitude of a drone acting as a LoRaWAN gateway to maximize coverage radius while minimizing power consumption.”

• GPT-4 produced a function using a propagation model and simple gradient descent to balance coverage vs. power.

• DeepSeek-V3 implemented a clean, vectorized solution with parameterization for different environments.

• Phi-4 defaulted to a simplistic linear search, effective but computationally inefficient.

• LLaMA-3.3 misinterpreted the prompt, producing a function optimizing for camera field of view instead of communication coverage.

This example underscores both the promise and pitfalls of relying on LLMs for specialized tasks.

Conclusion

The comparative study demonstrates that DeepSeek-V3 and GPT-4 remain the most reliable LLMs for LoRaWAN-related engineering tasks, consistently generating correct and executable Python code. However, the emergence of models like Phi-4 and LLaMA-3.3 shows that lightweight alternatives are not only viable but also surprisingly competent when carefully prompted.

For practitioners, the takeaway is clear:

• Use heavyweight models for guaranteed accuracy and complex optimization.

• Use lightweight models for cost-efficient, offline, or field-deployable scenarios, provided there is tolerance for minor errors and debugging.

Ultimately, the integration of LLMs into engineering workflows is not about replacing experts but about augmenting their capabilities, accelerating prototyping, and lowering barriers to advanced computation. With proper model selection, prompt engineering, and domain-specific tuning, LLMs hold tremendous promise for the future of IoT and wireless engineering.