Recent Posts
Log Record Counts in PySpark (With a Timer)
When I’m working with large datasets in PySpark, I often need to know how many records are flowing through my transformations. It’s a simple thing, but being able to log that information at the right time can help me catch issues early—like unexpected filters wiping out rows or joins ballooning in size. What makes it
Balancing Git Commit Quantity and Quality
Commit Quantity vs Quality Balancing commit quantity and quality is a common challenge for software engineers. While some emphasize frequent, incremental commits, others advocate for single, clean commits that ensure a pristine history. In this post, I’ll share my personal approach to navigating this balance, lessons from past experiences, and tips for fostering a team
Day One: Starting Strong as a New Software Engineer
Welcome to your first day as a software engineer! Whether you’re joining your first tech company or transitioning into a new role, the beginning can feel like a mix of excitement, nerves, and information overload. But don’t worry—there are a few key things you can do on Day One to set the tone for success
Fixing PySpark Import Errors When Using Custom Modules
Encountering a ModuleNotFoundError while running a PySpark job can be frustrating, especially when the module exists and works perfectly outside of Spark. I recently ran into this exact issue while working with a Spark job that imported a function from a local utils module. Everything seemed fine until I tried to use a PySpark UDF,
Running Development Mode in Multiple NPM Workspaces
When working on a monorepo with multiple packages, getting everything up and running efficiently can be a challenge. I’ve been there—jumping between terminal windows, manually starting different services, and hoping I didn’t forget anything. Fortunately, NPM workspaces provide a great way to manage multiple packages in a single repository, and with the right setup, starting
Building a React File Upload Button to Parse JSON and YAML
Handling file uploads in a React app is straightforward, but things get more interesting when you need to filter for specific file types and parse their contents. I needed to create an upload button that only accepted JSON and YAML files, then read and displayed their contents in the browser. Here’s how I did it.
