A high-level, interpreted programming language known for its simplicity and readability, widely used for web development, data analysis, and automation.
Encountering a ModuleNotFoundError while running a PySpark job can be frustrating, especially when the module exists and works perfectly outside of Spark. I recently ran into this exact issue while working with a Spark job that imported a function from a local utils module. Everything seemed fine until I tried to use a PySpark UDF,
I’ve spent a lot of time refining my Python code, and one technique that has greatly improved my projects is the use of the built-in super() function. In my early days, I used to hardcode parent class names when calling inherited methods. Over time, I realized that super() not only simplifies the code but also
Handling sensitive information is a critical aspect of software development, especially when dealing with user data. Whether it’s masking Social Security Numbers (SSNs), email addresses, or URLs, ensuring that this data is appropriately scrubbed is essential for maintaining privacy and security. In this post, I’ll share my experience creating a Python function that efficiently scrubs
Flink’s Kafka integration can be an excellent choice for building near real-time data pipelines. Offsets are at the center of these pipelines, governing exactly where Flink should begin reading data and how it responds to any missing or invalid positions. By combining Flink’s offset initializers with Kafka’s own offset resets, you can create robust and
Automating tests is a crucial part of any development workflow, ensuring that your code remains reliable and free of regressions. In my journey as a developer, I’ve often found the need to rerun tests automatically whenever I make changes to my files. This not only saves time but also provides immediate feedback, allowing me to
I knew from the start that Microsoft’s Visual C Runtime Library (MSVCRT) would not be natively available on macOS, so I wasn’t surprised when I realized it wouldn’t install. Because I don’t often rely on MSVCRT, it didn’t feel like a critical loss—until a coworker asked me to make a shared Python script run smoothly
When I was working on debugging a PyFlink job running inside a Docker container, I encountered a significant challenge: accessing and monitoring logs directly from my host machine. Traditional logging methods proved cumbersome, making it difficult to trace issues effectively. To overcome this hurdle, I decided to implement a custom Python logger that sends log
Testing is one of those steps in software development that separates a hobby project from a professional one. Whether you’re building something small or maintaining a complex codebase, testing ensures your code does what it’s supposed to do. If you’re new to testing in Python, pytest is one of the easiest tools to get started
Have you ever run into the frustrating problem of your script not finding the file it’s supposed to read? I know I have, especially when moving a project between environments or sharing it with a teammate. One key lesson I’ve learned is to always rely on relative paths that adapt to the script’s location. This
When I first stumbled upon the need to show differences between strings in a Git-like format, I wasn’t sure where to start. The idea seemed daunting—how could I highlight additions, removals, and changes in a clear, color-coded way? Luckily, Python’s difflib module came to the rescue. What started as a challenge turned into a rewarding
When I first started working with Python, one of the most common tasks I encountered was searching for a specific string in a list (or array) of strings. Whether it was looking up a username, checking for keywords, or validating input, this task came up over and over again. Thankfully, Python offers several easy and
Working with data in Python often means dealing with various file formats, and JSONL (JSON Lines) is one of the most practical for handling structured data in bulk. If you’ve come across this format and wondered how to read it efficiently, you’re in the right place. JSONL files are a series of JSON objects, each