DataFlow
OpenDCAI/DataFlow is a tool developed for data preparation and training. It's intended to generate, refine, evaluate and filter high-quality data for AI from noisy sources such as PDFs, plain text, and low-quality QA. This tool aims to improve the performance of large language models (LLMs) thr...
DataFlow.
OpenDCAI DataFlow is an open-source framework designed to streamline data preparation and training workflows for AI applications. It specializes in generating, refining, evaluating and filtering high-quality datasets from noisy sources including PDFs, plain text files, and low-quality question-answer pairs. The tool aims to improve large language model performance by providing clean, well-structured training data through automated data processing pipelines.