Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

1. The Problem: Multi-Task in a PEFT World

Large code models (LCMs) like CodeLlama, DeepSeek-Coder, and Qwen2.5-Coder have made impressive progress on individual code-related tasks, but adapting them to a specific downstream setting still requires fine-tuning. Full fine-tuning is expensive and storage-heavy — one full copy of the model per task — and parameter-efficient fine-tuning (PEFT) methods like QLoRA have emerged as the practical alternative for the single-task case.

But real software engineering workflows are not single-task. A useful coding assistant generates code from a description, summarizes existing functions, translates code between languages, and switches between these modes inside the same session. The interesting question is whether a single QLoRA-fine-tuned model can handle multiple code tasks simultaneously — and not just on functional correctness, but also on the non-functional qualities (complexity, maintainability, code smells) that determine whether the produced code is actually deployable.

To our knowledge, this is one of the first studies to jointly evaluate functional correctness and non-functional code quality in multi-task QLoRA settings. Most prior PEFT work focused either on a single task at a time, or on functional metrics only. We study QLoRA as one representative method from the LoRA family; we do not test other PEFT families such as adapters or prompt and prefix tuning.

2. What We Did: 28 Models, 3 Tasks, 4 Scales

We took the Qwen2.5-Coder-Instruct family at four scales — 0.5B, 1.5B, 3B, and 7B parameters — and ran a controlled comparison across three training configurations and three code-related tasks. To check that the RQ1 results are not tied to one model family, we repeated RQ1 on CodeLlama-7B-Instruct with the same data, adapter settings, hyperparameters, and evaluation.

That gives 28 trained models. The Qwen2.5-Coder grid has 20: MT-QLoRA and MT-FFT each yield one multi-task model per scale (4 + 4 = 8), and ST-QLoRA yields one model per task per scale (3 tasks × 4 scales = 12). CodeLlama-7B adds 4 (1 MT-QLoRA + 3 ST-QLoRA). The last 4 are 3B QLoRA ablation models: one trained with balanced task sampling and three trained on two-task subsets. Together: 20 + 4 + 4 = 28.

Datasets

Training and validation data come from CodeXGLUE. For code generation we inverted the Code-to-Text pairs (docstring → code); summarization uses the same pairs in the normal direction; translation uses the CodeXGLUE Java↔C# pairs. For testing, code generation uses CoderEval problems with verified test suites, summarization uses a curated CoderEval subset of long functions (from Crupi et al.), and translation uses the CodeXGLUE test split. Table 2 gives the counts.

Table 2 — Dataset Statistics

Task	Language	Train	Validation	Test
Code Summarization	Java	164,923	5,183	99*
	Python	251,820	13,914	100*
Code Generation	Java	164,923	5,183	184*
	Python	251,818	13,914	190*
Code Translation	Java → C#	10,300	500	1,000
	C# → Java	10,300	500	1,000

Table 2. Dataset statistics. Train and validation sets are from CodeXGLUE. * Test instances from CoderEval.

Metrics

We evaluated along four axes:

• Functional correctness — pass@1 on CoderEval (execution-based) plus CodeBLEU for code generation; CodeBLEU alone for translation, since the CodeXGLUE translation data has no tests.
• Summary similarity — BLEU, METEOR, ROUGE-L, chrF, BERTScore, and SIDE (Java only).
• Non-functional code quality — static analysis with Lizard (lines of code, tokens, cyclomatic complexity, detection rate), Pylint (Python), PMD (Java), Roslyn (C# syntax errors and maintainability issues), and SonarCloud (security hotspots, reliability, maintainability, cognitive complexity).
• LLM-as-a-judge — GPT-5 Mini rates each summary for content adequacy, conciseness, and fluency on a 1–5 scale, five times, and we average the runs. We checked the judge against two human annotators on 200 summaries from the 3B models: Pearson correlation with the mean human score was 0.633 for content adequacy, 0.827 for conciseness, and 0.877 for fluency. The judge scores content adequacy about 0.8 points lower than the humans do, but keeps their ranking.

For statistical reliability, paired comparisons used McNemar's test (binary outcomes such as pass@1) and the Wilcoxon signed-rank test (continuous metrics), with Holm-Bonferroni correction for multiple comparisons.

3. RQ1 — Multi-Task QLoRA vs Single-Task QLoRA

Does training one QLoRA model on three tasks at once cost us anything compared to training a separate QLoRA per task? The discussion below covers 0.5B–3B; 7B and CodeLlama-7B are in the tables and summarized at the end of the section.

Code Generation

Up to 3B, MT-QLoRA is competitive with ST-QLoRA, and its edge tends to grow with model size. Java at 1.5B is the clear exception: 19.02% vs 25.54% pass@1, a 25.5% relative drop.

Language	Model	MT-QLoRA pass@1	ST-QLoRA pass@1	Direction
Python	0.5B	15.79%	15.79%	Same
	1.5B	18.95%	16.84%	MT > ST
	3B	21.05%	20.53%	MT > ST
	7B	20.53%	22.63%	ST > MT
	CodeLlama-7B	21.58%	21.58%	Same
Java	0.5B	20.11%	21.74%	ST > MT
	1.5B	19.02%	25.54%	ST > MT
	3B	32.07%	29.89%	MT > ST
	7B	33.70%	30.43%	MT > ST
	CodeLlama-7B	25.00%	33.15%	ST > MT

RQ1 — pass@1 for code generation, MT-QLoRA vs ST-QLoRA (Tables 6 and 7). Rows without a family name are Qwen2.5-Coder.

Static analysis follows the same scale pattern. Smaller MT models produce more quality issues: Python SonarCloud maintainability issues rise from 53 to 90 at 0.5B and from 45 to 72 at 1.5B. At 3B the trend reverses: for Python, MT-QLoRA has 33.8% fewer maintainability issues (106 vs 160) and 19.3% lower cognitive complexity (549 vs 680) than ST-QLoRA. Summed over 0.5B–3B, Python MT code has slightly more issues (cyclomatic complexity +7.1%), while Java MT code has slightly fewer (PMD −2.4%, SonarCloud −4.0%). McNemar (for pass@1) and Wilcoxon (for quality metrics) tests with Holm-Bonferroni correction found no significant difference at any scale from 0.5B to 3B in either language, so the safe summary is: MT-QLoRA matches ST-QLoRA on correctness, and larger MT models balance correctness and quality better.

Code Translation

Translation is more sensitive to direction and scale (CodeBLEU, Tables 8 and 9).

• Java→C#: MT-QLoRA trails ST-QLoRA at 0.5B (68.92% vs 75.77%) and 3B (62.88% vs 66.83%) and is slightly ahead at 1.5B (73.08% vs 72.27%). MT-QLoRA produces more compact C# at every scale (2.3–4.8% fewer lines). Roslyn issues swing widely: 66.8% fewer than ST-QLoRA at 1.5B, but 137.0% more at 3B.
• C#→Java: again 1.5B is the sweet spot (70.72% vs 69.35%); MT-QLoRA trails at 0.5B (68.37% vs 75.36%) and 3B (71.22% vs 72.81%). At 3B, MT-QLoRA still reduces PMD issues by 13.8% and SonarCloud issues by 22.7%.

So translation has no consistent winner. Gains in one quality signal (fewer lines, lower complexity) can sit next to regressions in another (Roslyn for C#). Wilcoxon tests find significant CodeBLEU differences for Java→C# at 0.5B, 1.5B, and 3B, but for C#→Java only at 0.5B, and effect sizes on quality metrics are small.

Code Summarization

Summarization shows a striking language-dependent effect.

Language	Model	BLEU (MT-QLoRA vs ST-QLoRA)	Direction
Python	0.5B	19.43% vs 14.29% (+36.0%)	MT > ST
	1.5B	22.90% vs 15.12% (+51.5%)	MT > ST
	3B	29.90% vs 23.31% (+28.3%)	MT > ST
	7B	30.80% vs 16.49% (+86.8%)	MT > ST
	CodeLlama-7B	30.89% vs 27.98%	MT > ST on most metrics; ST leads on ROUGE-L and BERTScore
Java	0.5B	7.18% vs 7.10%	Mixed — MT ahead on BLEU and ROUGE-L, behind on METEOR, chrF, BERTScore, SIDE
	1.5B	8.12% vs 8.48%	ST > MT on all metrics but SIDE
	3B	7.63% vs 8.73%	ST > MT on every metric
	7B	8.37% vs 8.19%	Mixed — split across metrics
	CodeLlama-7B	10.96% vs 8.50%	MT > ST on most metrics

RQ1 — code summarization (Table 10). Python favors multi-task training at every scale and in both families; Java favors single-task training from 1.5B to 3B, but not at 7B or for CodeLlama.

The LLM-as-a-judge evaluation (GPT-5 Mini) tells a slightly more nuanced story: for Java summarization, no significant quality differences between MT-QLoRA and ST-QLoRA, while for Python there is no difference at 0.5B but MT-QLoRA is significantly better on all three dimensions at 1.5B and 3B.

7B and CodeLlama-7B

At 7B the picture splits by task. Code generation diverges by language: MT-QLoRA leads on Java (33.70% vs 30.43%) but trails on Python (20.53% vs 22.63%), a reversal of the 3B result for Python. Python summarization still favors MT-QLoRA by a wide margin (BLEU 30.80% vs 16.49%). Translation breaks: ST-QLoRA beats MT-QLoRA by 40.3% on Java→C# (77.70% vs 55.37%) and 43.0% on C#→Java (71.26% vs 49.83%), a much bigger gap than at 3B. The cause is left open; one candidate the paper names is that the adapter size is fixed, so the trainable share falls to 0.27% at 7B, while translation has by far the smallest share of the training data.

On CodeLlama-7B, the translation and Python summarization results repeat: ST-QLoRA wins translation in both directions (CodeBLEU 82.22% vs 79.45% on Java→C#, 82.48% vs 79.98% on C#→Java), and MT-QLoRA wins Python summarization on most metrics. Code generation does not repeat: pass@1 is tied on Python and ST-QLoRA leads on Java (33.15% vs 25.00%), even though CodeLlama's MT-QLoRA Java code is cleaner (PMD −20.7%, SonarCloud −20.4%). None of the CodeLlama differences is statistically significant.

4. RQ2 — Multi-Task QLoRA vs Multi-Task Full Fine-Tuning

The next question is whether the parameter-efficient route gives anything up relative to multi-task full fine-tuning.

Code Generation

For both Python and Java, MT-QLoRA matches or exceeds MT-FFT in most configurations, and the gap grows with scale. Java at 1.5B is the one case where MT-FFT is ahead. McNemar tests find no significant pass@1 difference from 0.5B to 3B, so on correctness the two are at parity there.

Table 11 — Python Code Generation: MT-QLoRA vs MT-FFT

Scale	MT-QLoRA pass@1	MT-FFT pass@1	Δ pass@1	Lizard / Pylint / SonarCloud
0.5B	15.79%	14.74%	+7.1%	CyC 812 vs 928 (−12.5%); Pylint −13.9%; SonarCloud maintainability −32.3%
1.5B	18.95%	15.79%	+20.0%	CyC 882 vs 928 (−5.0%); slightly more Pylint issues
3B	21.05%	17.37%	+21.2%	32.0% lower CyC (677 vs 996); Pylint −9.0%; SonarCloud maintainability −21.5%; 35.0% fewer lines
7B	20.53%	14.74%	+39.3%	CyC 732 vs 782

Table 11. Python code generation, MT-QLoRA vs MT-FFT. MT-QLoRA has higher pass@1 at every scale and much less complex code at 3B.

Table 12 — Java Code Generation: MT-QLoRA vs MT-FFT

Scale	MT-QLoRA pass@1	MT-FFT pass@1	Δ pass@1	Lizard / PMD / SonarCloud
0.5B	20.11%	19.02%	+5.7%	CyC 697 vs 649 (+7.4%); fewer PMD issues
1.5B	19.02%	20.11%	−5.4%	CyC 589 vs 676 (−12.9%); 18.5% fewer lines; fewer PMD issues
3B	32.07%	26.63%	+20.4%	CyC 558 vs 607 (−8.1%); fewer PMD and SonarCloud issues
7B	33.70%	18.48%	+82.4%	CyC 663 vs 778

Table 12. Java code generation, MT-QLoRA vs MT-FFT. MT-FFT leads only at 1.5B; MT-QLoRA is far ahead at 3B and 7B. Summed over 0.5B–3B, MT-QLoRA has 6.3% fewer PMD violations.

Code Translation

Translation mostly favors full fine-tuning (Tables 13 and 14):

• Java→C#: MT-FFT leads on CodeBLEU at every scale (74.91% vs 68.92% at 0.5B, 78.53% vs 73.08% at 1.5B, 66.38% vs 62.88% at 3B), and the differences are significant (with a negligible effect size). MT-QLoRA also has 40.9% more Roslyn issues summed over 0.5B–3B.
• C#→Java: MT-FFT leads at 0.5B and 3B; MT-QLoRA is slightly ahead at 1.5B (70.72% vs 70.07%). At 3B, MT-QLoRA code is much cleaner: SonarCloud issues drop by 69.6% (435 vs 1,430), and PMD issues are 18.7% lower summed over 0.5B–3B.
• 7B: MT-FFT leads by 30.2% on Java→C# (72.10% vs 55.37%) and 41.6% on C#→Java (70.54% vs 49.83%), against gaps of at most 8% at the smaller scales.

Code Summarization

The biggest gap between MT-QLoRA and MT-FFT shows up in Python summarization, where MT-QLoRA far outperforms MT-FFT on BLEU at every scale.

Language	Scale	BLEU (MT-QLoRA vs MT-FFT)	Direction
Python	0.5B	19.43% vs 9.53% (+103.9%)	MT-QLoRA > MT-FFT
	1.5B	22.90% vs 12.67% (+80.7%)	MT-QLoRA > MT-FFT
	3B	29.90% vs 22.54% (+32.7%)	MT-QLoRA > MT-FFT
	7B	30.80% vs 9.76% (+215.6%)	MT-QLoRA > MT-FFT
Java	0.5B	7.18% vs 7.70%	Mixed — MT-QLoRA better on METEOR, ROUGE-L, SIDE; MT-FFT better on BLEU, chrF, BERTScore
	1.5B	8.12% vs 7.43%	Mixed — MT-QLoRA better on all metrics but BERTScore
	3B	7.63% vs 7.90%	Mixed — MT-QLoRA better on all metrics but BLEU
	7B	8.37% vs 6.77%	Mixed — close overall; MT-QLoRA better on all metrics but SIDE

RQ2 — Code summarization (Table 15). Python heavily favors MT-QLoRA over MT-FFT; Java is metric-dependent.

The LLM judge shrinks these gaps. For Java, judge scores show no significant difference between MT-QLoRA and MT-FFT. For Python, MT-QLoRA is significantly better on all three dimensions, but the absolute scores are close: at 0.5B, BLEU rises 103.9% while content adequacy barely moves (1.818 vs 1.806).

Training Cost

MT-QLoRA also trains far fewer parameters and needs less GPU memory. The adapter configuration is the same at every scale, so the trainable share falls as the model grows.

Scale	Peak GPU memory, MT-FFT	Peak GPU memory, MT-QLoRA	Trainable params, MT-FFT	Trainable params, MT-QLoRA	Reduction
1.5B	13.04 GB	10.48 GB	1.54B (100%)	9.2M (0.60%)	167×
3B	23.28 GB	15.68 GB	3.09B (100%)	15.0M (0.49%)	206×
7B	39.92 GB	24.15 GB	7.62B (100%)	20.2M (0.27%)	377×

Table 16. Training cost of MT-FFT and MT-QLoRA (reported for 1.5B, 3B, and 7B).

5. Key Takeaways

• Multi-task QLoRA is not a cost-saving compromise for generation and summarization. At 1.5B, 3B, and 7B it is competitive with or better than both single-task QLoRA and multi-task full fine-tuning on most of these comparisons, while training 0.27–0.60% of the parameters. Translation is the exception.
• Bigger helps, up to a point. Larger models balance correctness and quality better; smaller models keep functionality but show more quality issues. At 7B the lead over MT-FFT grows for code generation (Java pass@1 33.70% vs 18.48%) and Python summarization, but translation falls well behind both ST-QLoRA and MT-FFT.
• Code quality holds up for generation. Against MT-FFT, MT-QLoRA code is often simpler and cleaner (32.0% lower Python cyclomatic complexity at 3B; 6.3% fewer Java PMD violations over 0.5B–3B). Against ST-QLoRA, generation quality differences are small and not statistically significant; in translation some are significant, and Roslyn results swing widely (−66.8% to +137.0%).
• Language-dependent effects exist. Python summarization benefits from multi-task training at every scale and on CodeLlama on most metrics; Java summarization favors single-task training from 1.5B to 3B, but not at 7B.
• Direction matters in translation. The Java→C# and C#→Java directions behave differently, 1.5B is the most consistent “sweet spot” for MT-QLoRA on translation up to 3B, and at 7B MT-QLoRA falls behind in both directions.
• The task mix matters. In 3B ablations, balanced task sampling narrows the Java→C# gap to ST-QLoRA from −5.9% to −1.1% but lowers C#→Java CodeBLEU by 7.9%. Dropping code generation from the mix breaks translation (Java→C# CodeBLEU falls to 5.35%), while training generation with translation alone gives the largest gains (73.71% Java→C# CodeBLEU, 36.41% Java pass@1).
• A second model family agrees in part. On CodeLlama-7B the translation and Python summarization results repeat; the code generation results do not.
• Correctness and quality together. To our knowledge, this is one of the first studies of multi-task QLoRA for code that evaluates functional correctness and non-functional code quality together — a more realistic standard for whether the produced code is actually deployable.

6. Bottom Line

A single QLoRA-optimized model can handle code generation, code translation, and code summarization at once, though translation is its weak spot, especially at 7B. For generation and summarization it mostly holds up against both per-task adapters and full fine-tuning, under static analysis and LLM-based evaluation, while training 0.27–0.60% of the parameters and using 60–80% of full fine-tuning's peak GPU memory. Our reading for practitioners: one MT-QLoRA model on a Qwen2.5-Coder-class base is a reasonable default for generation and summarization; for translation, compare it against single-task adapters or full fine-tuning first, since MT-QLoRA trails in most translation settings and falls far behind at 7B. Java generation at 0.5B and 1.5B and Java summarization from 1.5B to 3B also favor single-task training.

For researchers, the more interesting message is methodological: evaluating multi-task PEFT on functional metrics alone gives an incomplete picture. Static analysis and LLM-as-a-judge change the picture in non-trivial ways, and BLEU can move a lot while judged quality barely changes. The results also cover QLoRA only, as one method from the LoRA family; whether they hold for other PEFT methods is open.