GitHub

pysparkdt (PySpark Delta Testing)

An open-source Python library for simplifying local testing of Databricks workflows using PySpark and Delta tables. This library enables seamless testing of PySpark processing logic outside Databricks ...
IEEE

Low-Frequency Magnetotelluric Data Denoising Using Improved Denoising Convolutional Neural Network and Gated Recurrent Unit

Abstract: The magnetotelluric (MT) signals are susceptible to anthropogenic noise and the existing denoising methods have significant shortcomings in low-frequency situations. To address the problem, ...
GitHub

A simple pytest plugin to run tests concurrently

This pytest plugin takes a set of tests that would be normally be run serially and execute them in parallel. The main goal of pytest-run-parallel is to discover thread-safety issues that could exist ...
IEEE

Evaluating LLM’s Code Reading Abilities in Big Data Contexts using Metamorphic Testing

Abstract: With the explosive growth of Big Data, understanding complex data-centering algorithms and software has become essential. Large Language Models (LLMs) especially ChatGPT models have been ...