OLMo: Accelerating the Science of Language Models

Abstract: Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, this technical report details the first release of OLMo, a state-of-the-art, truly Open Language Model and its framework to build and study the science of language modeling. Unlike most prior efforts that have only released model weights and inference code, we release OLMo and the whole framework, including training data and training and evaluation code. We hope this release will empower and strengthen the open research community and inspire a new wave of innovation.


Teachers’ Perceptions of Literacy Implementation in Elementary Schools

Abstract: The purpose of this research is to find out how teachers perceive the implementation of literacy at the Gugus 1 Elementary School, North Padang District, Padang City. The research method used is quantitative descriptive. Data collection techniques through distribution of surveys implementing literacy questionnaires. The dimensions of implementing literacy described in this research consist of positive statements and negative statements. The results of the research show that teachers have a fairly positive perception tendency, this can be seen from the percentage of scores obtained from distributing the questionnaire and the explanation from the table. A very positive perception shows that teachers also participate in literacy activities at school. These results come from an analysis of the factors that cause teacher perceptions, namely functional factors and structural factors. Apart from that, it can be said that the teacher has been successful in implementing reading literacy.


FedAnchor: Enhancing Federated Semi-Supervised Learning with Label  Contrastive Loss for Unlabeled Clients

Abstract: Federated learning (FL) is a distributed learning paradigm that facilitates collaborative training of a shared global model across devices while keeping data localized. The deployment of FL in numerous real-world applications faces delays, primarily due to the prevalent reliance on supervised tasks. Generating detailed labels at edge devices, if feasible, is demanding, given resource constraints and the imperative for continuous data updates. In addressing these challenges, solutions such as federated semi-supervised learning (FSSL), which relies on unlabeled clients' data and a limited amount of labeled data on the server, become pivotal. In this paper, we propose FedAnchor, an innovative FSSL method that introduces a unique double-head structure, called anchor head, paired with the classification head trained exclusively on labeled anchor data on the server. The anchor head is empowered with a newly designed label contrastive loss based on the cosine similarity metric. Our approach mitigates the confirmation bias and overfitting issues associated with pseudo-labeling techniques based on high-confidence model prediction samples. Extensive experiments on CIFAR10/100 and SVHN datasets demonstrate that our method outperforms the state-of-the-art method by a significant margin in terms of convergence rate and model accuracy.


A Class Specific Feature Selection Method for Improving the Performance of Text Classification

Abstract: Recently, a significant amount of research work has been carried out in the field of feature selection. Although these methods help to increase the accuracy of the machine learning classification, the selected subset of features considers all the classes and may not select recommendable features for a particular class. The main goal of our paper is to propose a new class-specific feature selection algorithm that is capable of selecting an appropriate subset of features for each class. In this regard, we first perform class binarization and then select the best features for each class. During the feature selection process, we deal with class imbalance problems and redundancy elimination. The Weighted Average Voting Ensemble method is used for the final classification. Finally, we carry out experiments to compare our proposed feature selection approach with the existing popular feature selection methods. The results prove that our feature selection method outperforms the existing methods with an accuracy of more than 37%.


The Importance of Interpretability in AI Systems and Its Implications for Deep Learning

Abstract: Particularly inside the context of deep learning, the concept of interpretability in artificial intelligence systems is crucial for boosting the degree of trust and self-belief that human beings have in machine-learning fashions. Deep learning models have many parameters and complex architectures that make them function like mysterious “black boxes,” making it difficult for users to apprehend how they function. This opacity increases questions about those models' ethics, dependability, and viable biases. In the field of deep learning, achieving interpretability is crucial for several reasons. First off, interpretable models enhance transparency by making the model's judgments and forecasts simpler for customers to understand. This is particularly essential in complicated fields like banking and healthcare, wherein knowledge and self-assurance are vital. Moreover, interpretability facilitates the identification and correction of biases in the model or the training statistics, performing as a car for fairness and duty.


Towards Spatially-Lucid AI Classification in Non-Euclidean Space: An  Application for MxIF Oncology Data

Abstract: Given multi-category point sets from different place-types, our goal is to develop a spatially-lucid classifier that can distinguish between two classes based on the arrangements of their points. This problem is important for many applications, such as oncology, for analyzing immune-tumor relationships and designing new immunotherapies. It is challenging due to spatial variability and interpretability needs. Previously proposed techniques require dense training data or have limited ability to handle significant spatial variability within a single place-type. Most importantly, these deep neural network (DNN) approaches are not designed to work in non-Euclidean space, particularly point sets. Existing non-Euclidean DNN methods are limited to one-size-fits-all approaches. We explore a spatial ensemble framework that explicitly uses different training strategies, including weighted-distance learning rate and spatial domain adaptation, on various place-types for spatially-lucid classification. Experimental results on real-world datasets (e.g., MxIF oncology data) show that the proposed framework provides higher prediction accuracy than baseline methods.


Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement  Learning

Abstract: Benchmarks play a crucial role in the development and analysis of reinforcement learning (RL) algorithms. We identify that existing benchmarks used for research into open-ended learning fall into one of two categories. Either they are too slow for meaningful research to be performed without enormous computational resources, like Crafter, NetHack and Minecraft, or they are not complex enough to pose a significant challenge, like Minigrid and Procgen. To remedy this, we first present Craftax-Classic: a ground-up rewrite of Crafter in JAX that runs up to 250x faster than the Python-native original. A run of PPO using 1 billion environment interactions finishes in under an hour using only a single GPU and averages 90% of the optimal reward. To provide a more compelling challenge we present the main Craftax benchmark, a significant extension of the Crafter mechanics with elements inspired from NetHack. Solving Craftax requires deep exploration, long term planning and memory, as well as continual adaptation to novel situations as more of the world is discovered. We show that existing methods including global and episodic exploration, as well as unsupervised environment design fail to make material progress on the benchmark. We believe that Craftax can for the first time allow researchers to experiment in a complex, open-ended environment with limited computational resources.


BETA-UAV: Blockchain-based Efficient Authentication for Secure UAV  Communication

Abstract: Unmanned aerial vehicles (UAV), an emerging architecture that embodies flying ad-hoc networks, face critical privacy and security challenges, mainly when engaged in data-sensitive missions. Therefore, message authentication is a crucial security feature in drone communications. This paper presents a Blockchain-based Efficient, and Trusted Authentication scheme for UAV communication, BETA-UAV, which exploits the inherent properties of blockchain technology concerning memorability and is immutable to record communication sessions via transactions using a smart contract. The smart contract in BETA-UAV allows participants to publish and call transactions from the blockchain network. Furthermore, transaction addresses are proof of freshness and trustworthiness for subsequent transmissions. Furthermore, we investigated their ability to resist active attacks, such as impersonation, replaying, and modification. In addition, we evaluate the gas costs associated with the functions of the smart contract by implementing a BETA-UAV on the Ethereum public blockchain. A comparison of the computation and communication overheads shows that the proposed approach can save significant costs over traditional techniques.


ICE-SEARCH: A Language Model-Driven Feature Selection Approach

Abstract: This study unveils the In-Context Evolutionary Search (ICE-SEARCH) method, the first work that melds language models (LMs) with evolutionary algorithms for feature selection (FS) tasks and demonstrates its effectiveness in Medical Predictive Analytics (MPA) applications. ICE-SEARCH harnesses the crossover and mutation capabilities inherent in LMs within an evolutionary framework, significantly improving FS through the model's comprehensive world knowledge and its adaptability to a variety of roles. Our evaluation of this methodology spans three crucial MPA tasks: stroke, cardiovascular disease, and diabetes, where ICE-SEARCH outperforms traditional FS methods in pinpointing essential features for medical applications. ICE-SEARCH achieves State-of-the-Art (SOTA) performance in stroke prediction and diabetes prediction; the Decision-Randomized ICE-SEARCH ranks as SOTA in cardiovascular disease prediction. Our results not only demonstrate the efficacy of ICE-SEARCH in medical FS but also underscore the versatility, efficiency, and scalability of integrating LMs in FS tasks. The study emphasizes the critical role of incorporating domain-specific insights, illustrating ICE-SEARCH's robustness, generalizability, and swift convergence. This opens avenues for further research into comprehensive and intricate FS landscapes, marking a significant stride in the application of artificial intelligence in medical predictive analytics.


Fake Job Post Detection using Machine Learning

Abstract: The proliferation of online job platforms has given rise to a concerning increase in fraudulent job postings, presenting significant risks to job seekers and undermining the credibility of the job market. This research paper aims to address the pressing issue of fake job post identification by leveraging machine learning techniques. The primary objective is to develop a robust automated tool capable of accurately distinguishing between authentic and deceptive job advertisements. The proposed methodology utilizes a range of machine learning algorithms, incorporating supervised learning techniques and natural language processing methods, to analyze and classify job postings. Through the integration of both single classifiers and ensemble classifiers, the system evaluates and compares results, effectively detecting fraudulent job postings on the web. The study underscores the need for a proactive approach, acknowledging the dynamic tactics employed by scammers. Continuous refinement and adaptation of the machine learning models are emphasized to stay ahead of evolving fraudulent strategies. Ultimately, this research contributes to establishing a more secure online job market, fostering trust among job seekers and mitigating the financial and emotional risks associated with deceptive job postings.