Abstract: This report synthesises findings from 15 peer-reviewed papers addressing the following research question: How does the correlation between human attention benchmarks and synthetic metrics vary across different types of multimodal models (e.g., vision-language models vs. pure visual models) on downstream. In this…

Abstract: This report synthesises findings from 10 peer-reviewed papers addressing the following research question: What is the difference in token efficiency and inference latency between DeepSeek-R1 and Claude when performing iterative code repair on FeedbackEval with full repository context. Recent generations of frontier…

Abstract: This report synthesises findings from 11 peer-reviewed papers addressing the following research question: What is the trade-off between accuracy and inference latency in DeepSeek-R1 versus baseline multimodal models on HumanEval-V when evaluated under memory-constrained environments. As Large Language Models (LLMs)…

Abstract: This report synthesises findings from 10 peer-reviewed papers addressing the following research question: How does the inference throughput of DeepSeek-R1 compare to Llama-2-70B on HumanEval across different batch sizes and hardware configurations. Quantization is a powerful tool for accelerating large language model…

Abstract: This report synthesises findings from 9 peer-reviewed papers addressing the following research question: How does the inference latency of DeepSeek-R1 compare to state-of-the-art autoregressive and non-autoregressive language models on HumanEval-V benchmarks when measured in tokens per second. Abstract The rapid…

Abstract: This report synthesises findings from 7 peer-reviewed papers addressing the following research question: To what extent does access to file-level context improve the robustness of DeepSeek-R1 and Claude against adversarial feedback loops in the FeedbackEval benchmark. As Large Language Models (LLMs) become…

Abstract: This report synthesises findings from 12 peer-reviewed papers addressing the following research question: How does cross-domain finetuning affect DeepSeek-V3's accuracy on GPQA Diamond compared to in-domain finetuning. As Large Language Models (LLMs) become increasingly integrated into secure software development…

Abstract: This report synthesises findings from 4 peer-reviewed papers addressing the following research question: How does the vulnerability classification accuracy of DeepSeek-R1 on the Big-Vul dataset correlate with its code repair success rate on SWE-bench Verified. Software defect detection is a critical task in software…

Abstract: This report synthesises findings from 11 peer-reviewed papers addressing the following research question: How does the inclusion of issue-specific file context affect the pass@1 accuracy of DeepSeek-R1 versus Claude on SWE-bench Verified compared to baseline context-free evaluations. As Large Language Models (LLMs)…

Abstract: This report synthesises findings from 10 peer-reviewed papers addressing the following research question: What is the inference efficiency trade-off when applying cross-domain finetuning to DeepSeek-V3 on GPQA Diamond tasks. We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total…

Abstract: This report synthesises findings from 10 peer-reviewed papers addressing the following research question: To what extent does cross-domain finetuning improve DeepSeek-V3's robustness to distribution shifts in GPQA Diamond questions. Abstract The rapid evolution of large language models (LLMs) has driven a…

Abstract: This report synthesises findings from 12 peer-reviewed papers addressing the following research question: How does Llama-3.1-8B's code generation performance on MBPP compare to other open-source 8B models like Falcon-8B and Mistral-8B in terms of pass@1 accuracy. Romanized Nepali, the Nepali language written in the…

Abstract: This report synthesises findings from 9 peer-reviewed papers addressing the following research question: What is the impact of vulnerability taxonomy alignment on the code repair success rates of DeepSeek R1 versus Codestral on the Big-Vul dataset. Many ML-based approaches have been proposed to automatically detect,…

Abstract: This report synthesises findings from 16 peer-reviewed papers addressing the following research question: What are the trade-offs between latency overhead and semantic preservation when applying GDPR-compliant anonymization techniques to Llama-3.1-8B inference pipelines. Large language models (LLMs) have achieved…

Abstract: This report synthesises findings from 15 peer-reviewed papers addressing the following research question: How do different code generation models scale in inference efficiency when evaluated on multilingual programming benchmarks like CodeMixBench. Large Language Models (LLMs) have achieved remarkable success in code…

Abstract: This report synthesises findings from 11 peer-reviewed papers addressing the following research question: How does the diversity-weight parameter in Vendi-RAG influence the robustness of FLAN-T5-xl against adversarial attacks (e.g., ANLI) in knowledge-intensive QA, and what is the correlation between. Machine…

Abstract: This report synthesises findings from 10 peer-reviewed papers addressing the following research question: How does the Performance-Efficiency Ratio vary across different inference budget thresholds for code generation tasks using models ranging from 0.5B to 13B parameters on HumanEval and MBPP benchmarks. We…

Abstract: This report synthesises findings from 1 peer-reviewed paper addressing the following research question: How does the diversity-weight parameter in Vendi-RAG affect its performance on the ELI5 dataset when using a sparse retriever versus a dense retriever, measured by ROUGE-L scores. Questa tesi affronta il problema…

Abstract: This report synthesises findings from 14 peer-reviewed papers addressing the following research question: How does adaptive diversity-weight tuning in Vendi-RAG affect throughput on the TriviaQA benchmark compared to fixed-weight retrieval for FLAN-T5-xxl, and what is the optimal efficiency-accuracy.…

Abstract: This report synthesises findings from 4 peer-reviewed papers addressing the following research question: How does the manifold-aware distance metric in DPR compare to Euclidean and cosine distance in terms of Recall@10 on Natural Questions (NQ) when the context window is limited to 512 tokens. The advent of…

Abstract: This report synthesises findings from 4 peer-reviewed papers addressing the following research question: Can Vendi-RAG's diversity-aware retrieval approach improve robustness against adversarial or out-of-domain questions in the ELI5 benchmark compared to BM25 and dense retrieval baselines. Large Language Models…

Abstract: This report synthesises findings from 3 peer-reviewed papers addressing the following research question: How does Vendi-RAG's performance scale with increasing document corpus size in terms of EM score and latency on the TriviaQA benchmark compared to traditional RAG. The rapid evolution of natural language…

Abstract: This report synthesises findings from 7 peer-reviewed papers addressing the following research question: How robust is Vendi-RAG's diversity optimization to domain shifts when evaluated on cross-domain benchmarks like TyDiQA and DROP with F1 score comparisons. Aligned large language models (LLMs) demonstrate…

Abstract: This report synthesises findings from 11 peer-reviewed papers addressing the following research question: How does the robustness of Llama-2-7B and Llama-3-8B in handling out-of-domain retrieval tasks compare when evaluated on MuSiQue with a constrained context window of 1024 tokens. Prompt engineering has emerged as…

Abstract: This report synthesises findings from 14 peer-reviewed papers addressing the following research question: How does the retrieval accuracy of Contriever and DPR encoders compare on the Natural Questions benchmark when the context window size is increased to 2048 tokens. Retrieval-Augmented Generation (RAG) has…