DaedTech

Stories about Software

Jan
15

By

Complain-Bragging Your Way to the Top

Category: The Life of a Programmer Tags: , | 4 Comments

First, I’d like to make a brief aside to re-introduce Amanda Muledy (@eekitsabug) who, in addition to editing and writing, is also an artist.  Lest you think that I have a single iota of drawing talent, any sketches that you see as post illustrations are her work, done specifically for the Daedtech blog.  Her drawings are certainly better than me scrounging images from Wikimedia Commons public domain, so I’m going to be using those as graphics whenever possible.

Mo Money, Mo Problems

You know what I hate? I hate it when I get caviar stains all over the leather in my brand-new, fully-loaded Porsche SUV. It’s even worse when it happens because I was distracted by all of the people emailing me to ask if I would review their code or be part of their startup or maybe just for my autograph. That really sucks.

FatCatOkay, so maybe none of those things applies to me, but you know what does? Being party to people couching shameless bragging and self promotion as complaints and impositions. I hate being rich and popular because blah-blah-blah-whatever-this-part-doesn’t-matter. In case you missed it, I’m rich and I’m popular.

Surely you know someone that engages in this sort of thing. It’d be nearly impossible not to because I think just about everyone does it from time to time and to some degree or another. Most people have the interpersonal acumen to realize that simply blurting out flattering facts about oneself is somewhat distasteful. However, finding some other context in which to mention those same facts is likely to alleviate some of that distaste.  And complaining somehow seems like an easy and immediate pretext for this alternate context. (The best way would probably be to get some sympathetic plant to lob softballs at you, or else to simply wait for someone else to mention your achievements, but those are elaborate and unreliable, respectively.)

When people complain-brag, it’s most naturally in regards to some subject with which both speaker and audience are familiar. In the weightroom, a weightlifter may complain-brag about how sore he is from bench-pressing 320 pounds yesterday. In the college world, a kid may complain-brag about how badly hungover he is from pounding twenty-eight beers last night. Some subjects, such as money, popularity, and achievement, are near universal, while others, like the examples in this paragraph, are domain specific. Another domain-specific and curious example of complain-bragging that I observe quite frequently is corporate complain-bragging (or, its more innocuous cousin, excuse-bragging).

Mo Meetings, Mo Problems

Think about the following phrases you probably hear (or utter) with regularity:

  1. “Man, it’s so hard to keep up with all of my email” == “A lot of people want or need to talk to me.”
  2. “Ugh, I’ve had so many meetings today that I haven’t been able to get a thing done.” == “I’m important enough to get included on a lot of meeting rosters.”
  3. “Sorry I’m late, but my 10:00 ran way over” == “I’m not really sorry that I’m late and I want you to know that I’m important enough to have back-to-back meetings and to keep you waiting.”
  4. “I have so many code review requests that I won’t get to write code for weeks!” == “I’m the gatekeeper, Plebe, and don’t you forget it.”

There’s an interesting power dynamic in the corporate world that came of age during the heyday of command-and-control style management via intimidation. That power dynamic is one in which managers sit in the seats of power and makers work at their behest. (For a primer on the manager/maker terminology and background on how they interact, see this wonderful post by Paul Graham.) Makers are the peons who work all day, contributing to the bottom line: factory workers, engineers, programmers, accountants… even salesmen, to an extent. Managers are the overhead personnel that supervise them. Makers spend their days making things and managers spend their days walking from one meeting room to the next, greasing the skids of communication, managing egos, adjudicating personnel decisions, and navigating company realpolitik minefields. Managers also make money, get offices, and hear “how high” when they tell the peons to jump.

It’s traditionally an enviable position, being manager, which leads many people to envy it.  And, in true fake-it-till-you-make-it style, this envy leads those same people to emulate it. So aspiring mid-level managers begin to act the part: dressing more sharply; saying things like, “synergy,” and, “proactive”; bossing other people around; and making themselves seem more manager-like, in general. And what do managers do throughout the day? Well, they shuffle around from meeting to meeting, constantly running late and having way too many emails to read. So what does someone wanting to look like a manager without looking like they’re trying to look like a manager do? Why, they complain-brag (and excuse-brag) in such a way as to mimic mid-level management. In effect, this complain-bragging, tardiness, and unresponsiveness becomes a kind of status currency within the organization with actual power capital being exchanged in games of chicken. Who can show up a few minutes late to meetings or blow off an email here and there? (Think I’m being cynical? Take a look around when your next meeting is starting late, and I’ll wager dollars to donuts that people show up in rough order of tenure/power/seniority, from most peon to most important.)

Rethinking the Goals

So what’s my point with all this? Believe it or not, this isn’t a tone-deaf or bitter diatribe against “pointy haired bosses” or the necessity of office politics (which actually fascinates me as a subject). A lot of managers, executives, and people in general aren’t complain-bragging at all when they assess situations, offer excuses, or make apologies. But intentional or not, managers and corporate power brokers over the course of decades have created a culture in which complain-bragging about busy-ness, multi-tasking, and “firefighting” are as common and culturally expected as inane conversations about the weather around the water cooler. Rather than complaining about this reality, I’m encouraging myself and anyone reading to avoid getting caught up in it quite so automatically.

Why avoid it? Well, because I’d argue that obligatory complain-bragging creates a mild culture of failure. Let’s revisit the bullet points from the previous section and consider them under the harsh light of process optimization:

  1. “Man, it’s so hard to keep up with all of my email” — If you’re getting that many emails, you’re probably failing to delegate and thus failing as a manager.
  2. “Ugh, I’ve had so many meetings today that I haven’t been able to get a thing done.” — You’re not getting anything done and business continues anyway, “so what is it you’d say… ya do here?”
  3. “Sorry I’m late, but my 10:00 ran way over” — You either don’t have control over your meetings or don’t have a clear agenda for them, so your 10:00 was probably a waste of time.
  4. “I have so many code review requests that I won’t get to write code for weeks!” — You’re a bottleneck and costing the company money. Fix this.

You see, the problem with corporate complain-bragging or complain-bragging in general is that it necessarily involves complaining, which means ceding control over a situation. After all, you don’t typically complain about situations for which you called all the shots and got the desired outcome. And sometimes these are situations that you ought to control or situations that you’d at least look better for controlling. So I’ll leave you with the following thought: next time you find yourself about to complain-brag according to the standard corporate script, make it a simple apology, a suggestion for improvement or, perhaps best of all, a silent vow to fix something. Do these, and other people might do your bragging for you.

Jan
13

By

JUnit Revisited

Category: Java Tags: , , |

Just as a warning, in this short post, I’m going to be writing unit tests that verify that primitives in Java do what they should and basically that gravity is still turned on. The reason for that is that I’d like to showcase some new Java unit testing goodies I’ve recently discovered since coming back into the Java fold a little here and there lately. I firmly believe that the more conversationally readable the contents of unit tests are, the more effective they will be at defining functional and internal requirements as well as showcasing the behavior of the system.

@Test
public void two_ints_are_equal() {
int x = 4;
int y = 4;
assertThat(x, is(y));
}

Coming from the .NET world and using MSTest, I’m used to semantics of Assert.AreEqual<int>(x, y) where, by convention, the “control” or expected value goes on the left and the actual value goes on the right. This is a compelling alternative in that it reads like a sentence, which is always good. The MSTest version reads “Are equal x and y” whereas this reads “x is y.” The less it reads like Yoda is talking, the better. So what enables this goodness?

import static org.junit.Assert.assertThat;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.CoreMatchers.not;
import static org.junit.matchers.JUnitMatchers.*;

The first import gives you assertThat(), obviously. assertThat() as shown above takes two parameters (there is an overload that takes a string as an additional parameter to let you specify a failure message): a generic type for the first parameter, and a “matcher” for the second parameter. Matchers perform evaluations on types and can be chained together in fluent fashion to allow construction of sentences that flow. For instance, you can chain the is() matcher and the not() matcher to get the following test:

@Test
public void two_ints_are_not_equal() {
int x = 4;
int y = 5;
assertThat(x, is(not(y)));
}

This really just scratches the surface and there are lots of additional matchers from hamcrest as well. You can even extend the functionality by defining your own matchers to cater to the ubiquitous language of the domain that you’re using. This just barely scratches the surface, but if you’re a java developer and haven’t given these a look, I’d suggest doing so. If you’re a .NET developer, it’s worth taking a peek at what’s going on elsewhere and perhaps defining your own such constructs if you’re feeling ambitious or looking for existing ones. In fact, if you know of good ones, please post ’em — I always like seeing what’s out there.

Jan
11

By

Programatically Filling Out PDFs in Java

Category: Java Tags: , , | 28 Comments

I just got done dealing with an interesting problem. I had one of those PDFs that’s a form you can fill out and was tasked with programatically filling it out. So, I busted out my google-fu and came across PDFBox. It’s a handy and fairly no-nonsense little utility not just for filling out forms, but for manipulating PDFs in general. I had no idea something like this existed (mainly because I’d never really thought about it).

I downloaded the jar for PDFBox and wrote a simple class to test out my theory. In setting up the class and poking around randomly in the documentation, I saw that the main object of interest was a PDDocument. So, I set about instantiating one and discovered that you needed to use something called a COSDocument, which took something called a RandomAccess (not the standard version of the file, but a special version from PDFBox), and then my eyes started to cross and I pulled back and discovered that this is really what I want:

private static PDDocument _pdfDocument;

private static void populateAndCopy(String originalPdf, String targetPdf) throws IOException, COSVisitorException {
	_pdfDocument = PDDocument.load(originalPdf);
	
	_pdfDocument.save(targetPdf);
	_pdfDocument.close();
}

Much easier. Now, as I got down to business of trying this out, I discovered via runtime exception that I needed two external dependencies. One was apache commons logging and the other was something called fontbox that was right there along with the PDFBox download, but I ignored in the beginning. Probably with this code alone you wouldn’t necessarily hit both of those problems, but you will eventually, so better to add those jars right up front.

So far I was able successfully to open a PDF and save it as another file, which isn’t exactly a new capability for any programming language with file I/O, so I added something a little more concrete to the mix:

private static void populateAndCopy(String originalPdf, String targetPdf) throws IOException, COSVisitorException {
	_pdfDocument = PDDocument.load(originalPdf);
	
	_pdfDocument.getNumberOfPages();
	
	setField("SomeFieldName", "SomeFieldValue");
	_pdfDocument.save(targetPdf);
	_pdfDocument.close();
}

Lo and behold, liftoff. I actually got the right number of pages in the document. Now I was getting somewhere. Time to get down to the real business. I did this by going to the “Cookbook” section of the project and seeing what was under form generation. Seeing that this just took me to the javadoc for examples, I went and downloaded the example code and pasted it into my project (modifying it to conform to the Egyptian-style braces. In this fashion, I had a method that would print out all of the fields in the PDF as well as a method that would let me set fields by name. When I ran the one that printed out all of the fields, I got a runtime exception about some deprecated method and I discovered that in the source code for that method, it just threw an exception. Presumably, the written examples predated some change that had deprecated that method — deprecated it with extreme prejudice!

Well, I’d like to say that I fought the good fight, but I didn’t. I just deleted the offending call since it was just writing to console. So here is the end result of that effort:

public class Populater {

	private static PDDocument _pdfDocument;
	
	public static void main(String[] args) {
		
		String originalPdf = "C:\\blah\\blah\\input.PDF";
		String targetPdf = "C:\\blah\\blah\\input.PDF";
		
		try {
			populateAndCopy(originalPdf, targetPdf);
		} catch (IOException | COSVisitorException e) {
			e.printStackTrace();
		}
		
		System.out.println("Complete");
	}

	private static void populateAndCopy(String originalPdf, String targetPdf) throws IOException, COSVisitorException {
		_pdfDocument = PDDocument.load(originalPdf);
		
		_pdfDocument.getNumberOfPages();
		//printFields();  //Uncomment to see the fields in this document in console
		
		setField("SomeFieldName", "SomeFieldValue");
		_pdfDocument.save(targetPdf);
		_pdfDocument.close();
	}
	
    public static void setField(String name, String value ) throws IOException {
        PDDocumentCatalog docCatalog = _pdfDocument.getDocumentCatalog();
        PDAcroForm acroForm = docCatalog.getAcroForm();
        PDField field = acroForm.getField( name );
        if( field != null ) {
            field.setValue(value);
        }
        else {
            System.err.println( "No field found with name:" + name );
        }
    }

    @SuppressWarnings("rawtypes")
	public static void printFields() throws IOException {
        PDDocumentCatalog docCatalog = _pdfDocument.getDocumentCatalog();
        PDAcroForm acroForm = docCatalog.getAcroForm();
        List fields = acroForm.getFields();
        Iterator fieldsIter = fields.iterator();

        System.out.println(new Integer(fields.size()).toString() + " top-level fields were found on the form");

        while( fieldsIter.hasNext()) {
            PDField field = (PDField)fieldsIter.next();
               processField(field, "|--", field.getPartialName());
        }
    }
    
    @SuppressWarnings("rawtypes")
	private static void processField(PDField field, String sLevel, String sParent) throws IOException
    {
        List kids = field.getKids();
        if(kids != null) {
            Iterator kidsIter = kids.iterator();
            if(!sParent.equals(field.getPartialName())) {
               sParent = sParent + "." + field.getPartialName();
            }
            
            System.out.println(sLevel + sParent);
            
            while(kidsIter.hasNext()) {
               Object pdfObj = kidsIter.next();
               if(pdfObj instanceof PDField) {
                   PDField kid = (PDField)pdfObj;
                   processField(kid, "|  " + sLevel, sParent);
               }
            }
         }
         else {
             String outputString = sLevel + sParent + "." + field.getPartialName() + ",  type=" + field.getClass().getName();
             System.out.println(outputString);
         }
    }
}

Going forward, I’ll certainly factor this into a new class and probably extract some methods and improve warning avoidance, but that’s the gist of it. This didn’t exactly take a long time, but it probably could have gone quicker if I’d known a little more up-front and had all example code in one place. Hopefully it helps you in that capacity.

By the way, if you liked this post and you're new here, check out this page as a good place to start for more content that you might enjoy.

Jan
6

By

Betrayed by Your Test Runner

Category: Other Languages Tags: , , , |

The Halcyon Days of Yore

I was writing some code for Apex this morning and I had a strange sense of deja vu. Without going into painful details, Salesforce is a cloud-based CRM solution and Apex is its proprietary programming language that developers can use to customize their applications. The language is sort of reminiscent of a stripped-down hybrid of Java and C#. The IDE you use to develop this code is Eclipse, armed with an Apex plugin.

The deja vu that I experienced transported me back to my college days working in a 200 oldschoollinuxlevel computer systems course where projects assigned to us were the kind of deal involving profs/TAs writing 95% of the code and we filled in the other 5%. I am always grateful to my alma mater for this since one of the things most lacking in university CS education is often concepts like integration and working on large systems. In this particular class, I was writing C code in pico and using a makefile to handle compiling and linking the code on a remote server. This generally took a while because of network latency, server business, it being 13 years ago, a lot of files to link, etc. The end result was that I would generally write a lot of code, run make, and then get up and stretch my legs or get a drink or something, returning later to see what had happened.

This is what developing in Apex reminds me of. But there’s an interesting characteristic of Apex, which is that you have to write unit tests, they have to pass, and they have to cover something like 70% of your code before you’re allowed to run it on their hardware in production. How awesome is that? Don’t you sometimes wish C# or Java enforced that on your coworkers that steal in like ninjas and break half the things in your code base with their checkins? I was pumped when I got this assignment and set about doing TDD, which I’ve done the whole time. I don’t actually know what the minimum coverage is because I’ve been at 100% the entire time.

A Mixed Blessing?

One of the first things that I thought while spacing out and waiting for compile was how much it reminded me of my undergrad days. The second thing I thought of, ruefully, was how much better served I would have been back then to know about unit tests or TDD. I bet that could have same me some maddening debugging sessions. But then again, would I have been better off doing TDD then? And, more interestingly, am I better off doing it now?

Anyone who follows this blog will probably think I’ve flipped my lid and done a sudden 180 on the subject, but that’s not really the case. Consider the following properties of Apex development:

  1. Sometimes when you save, the IDE hangs because files have to go back to the server.
  2. Depending on server load, compile may take a fraction of a second or up to a minute.
  3. It is possible for the source you’re looking at to get out of sync with the feedback from the compiling/testing.
  4. Tests in a class often take minutes to run.
  5. Your whole test suite often takes many, many minutes to run.
  6. Presumably due to server load balancing, network latency and other such factors, feedback time appears entirely non-deterministic.
  7. It’s normal for me to need to close Eclipse via task manager and try again later.

Effective TDD has a goal of producing clean code that clearly meets requirements at the unit level, but it demands certain things of the developer and the development environment.  It is not effective when the feedback loop is extremely slow (or worse, inaccurate) since TDD, by its nature, requires near constant execution of unit tests and for those unit tests to be dependable.

Absent that basic requirement, the TDD practitioner is faced with a conundrum.  Do you stick to the practice where you have red (wait 2 minutes), green (what was I doing again, oh yeah, wait 3 minutes), refactor (oops, I was reading reddit and forgot what I was doing)?  Or do you give yourself larger chunks of time without feedback so that you aren’t interrupted and thrown out of the flow as often?

My advice would be to add “none of the above” to the survey and figure out how to make the feedback loop tighter.  Perhaps, in this case, one might investigate a way to compile/test offline, alter the design, or to optimize somehow.  Perhaps one might even consider a different technology.  I’d rather switch techs than switch away from TDD, myself.  But in the end, if none of these things proves tenable, you might be stuck taking an approach more like one from 20+ years ago: spend a big chunk of time writing code, run it, write down everything that went wrong, and trying again.  I’ll call this RDD — restriction driven development.  I’d say it’s to be avoided at all costs.

I give force.com an A for effort and concept in demanding quality from developers, but I’d definitely have to dock them for the implementation since they create a feedback loop that actively discourages the same.  I’ve got my fingers crossed that as they expand and improve the platform, this will be fixed.

Jan
3

By

Practical Programmer Math: From Booleans to Bits to Bytes

Category: Practical Math Tags: , |

The “Why This Should Matter to You” Story

A few scenarios today:

  1. You don’t understand why the answer to “how many bytes are in a kilobyte” isn’t 1000 (at least, not exactly).
  2. You see if((x & mask) > 7) and wonder why it compiles despite the author forgetting an ‘&’
  3. You feel a little ashamed because in spite of the fact that your non-programmer friends say that you “think in ones and zeroes”, you don’t actually understand how your code becomes ones and zeroes.

Everything here is symptomatic of a lack of understanding of the way data is represented at the most fine grained level in computer science and in programming.
It’s possible to understand some programming concepts without understanding the mechanics of how they work internally. After all, you can understand that a list.sort() method without understanding the sort algorithm being used.

So the general story is that your understanding of the data and data structures that go into programming breaks down at some level. You may understand that a string is a series of characters, but perhaps not that characters are actually integers or that integers are actually bytes or that bytes are actually series of bits or that bits are simply Booleans. And that breakdown in understanding somewhere in the chain leads to situations where you don’t fully understand what your peers are talking about.

Math Background

In the last two posts in this series, I’ve gone through some basic but important points about Boolean algebra, which one might describe as the basic mathematics of truth values. In this system there are two fundamental values to which constants, variables and expressions might evaluate: true and false. Boolean algebra is, in a very real sense, the absolute fundamental building block for everything that we as programmers do, but it may not be entirely obvious as to how.

To get there, let’s leave behind the syntax of Boolean algebra and pick up the syntax of machine language. So forget about “true” and “false” and consider instead 1 and 0, respectively. 1 is true, 0 is false. And further let’s get rid of the conjunction, disjunction and negation operators and consider, in their stead, the more familiar & | ~ operators. (You’ll notice that these are single operators and not the double operators with which you are likely familiar and this is not a typo — there is a method to my madness.) All of the rules from the previous posts still apply so, for instance, the identity law of A | 0 = A applies.

So far we’ve just swapped syntactical operators and constants. But let’s now consider an interesting property of basic arithmetic (not Boolean arithmetic) which is that written numbers are really a sequence of answers to a pre-arranged sequence of questions.

Wat

Okay, let me clarify with an example. Let’s pick a three digit number like 481. There’s nothing particularly special about this number, but let’s think of it in terms of a Q&A session. It goes like this “how many hundreds are in this number?” followed by the answer “four.” “How many tens are in this number?” “Eight.” “And how many ones?” “One.” This is really all there is to numerical representation of natural numbers (discounting the concept of zero, negative numbers, decimals, etc).

In all numbers that we’re comfortable with in normal society, the question always involves powers of 10: 1, 10, 100, etc. And so the answer is always a number 0 through 9. But what if we asked the question in a different way? What if we did it in terms of 7? Or 34? Or… 2? The answer, respectively, would always be an answer 0 through 6, 33… and 1. That last one is most interesting to us here. The answer will always be between 0 and 1 or, put another way, the answer will always be 0 or 1. Interesting.

Let’s look at an example. The number 10110, using powers of 2, is asking the question, “how many 16’s, 8’s, 4’s, 2’s and 1’s are there” to which the answers come as “1, 0, 1, 1, 0” respectively. We have 1 sixteen, no eights, 1 four, 1 two and no ones for a total of 22. 10110 is the binary representation of 22. That’s probably not new to you regardless of your CS/math/etc background if you’re a programmer. But what might be new to you is just how fundamental this concept is because it allows translation between physical and “soft”-ware concepts. How so? Because you know what’s actually pretty hard to represent and measure physically? 0 through 9. But you know what isn’t? Booleans. On or off. There or not. Current flowing or nothing. Chad punched or not. Magnetic surface raised or flat. Ion charged or neutral. You get the idea.

We can build computers where information is stored as a series of Booleans representing not “true or false”, but “yes or no” and we can still apply all of the same principles, rules, equivalences and concepts, albeit with slightly different semantics. We can take these “yes or no” values and assemble them into integers using our binary question schemes. We can them assemble those integers into ASCII characters, strings, basic programs, complex programs, multi-media files, and you know the rest.

The math behind all of programming is simple. The basis for the whole thing is simply gigantic sequences of Boolean values assembled into ever-larger pieces. The marriage of Boolean and counting unique to binary encoding means that you know how to assemble the Boolean sequences into meaningful information and also how to manipulate information and store it back, performing operations using identity, domination, negation, and other equivalences (we will get to some interesting optimizations using bit-shifting and other techniques in future posts).

The only missing piece here is how we get from these individual “1 or 0” values, called “bits” to larger values. Well, most of this is arbitrary. For instance, the definition of “byte” is simply “8 consecutive bits,” meaning that a byte can represent one of 256 possible integer values. These different values can also be assigned (again arbitrarily) to non-integral values such as characters. This is the basis for ASCII, a near-universal standard for representing text using integers (and thus binary/bits). This is the mechanism by which the building blocks are assembled.

How it Helps You

In a (hyphenated) word: background-knowledge. For instance, with all of this in mind, it’s relatively easy to get to solutions for the mysteries at the beginning of the post. First up is the weirdness surrounding bit counting using KB, MB, GB, etc. Is a kiloybyte a thousand bytes the way a kilogram is a thousand grams? Well, it could be, if you define it the way we’re used to seeing those terms defined. But that isn’t actually what winds up happening. What happens instead is the result of an interesting quirk of powers of two. As it turns out, 2 to the 10th is 1,024, which is roughly a ‘kilo’ of bytes. 2 to the 20th is 1,048,576, which is roughly a ‘mega’ (1 million) of bytes. Same kind of “fuzzy” counting happens with Giga, Tera, etc, all the way up the chain with 2 to the 30th, 40th, etc. So why reuse a prefix that means something else — something otherwise very precise? Personally, I think the adopters of this convention were being cute, and I don’t care for it at all (*dons flame retardant suit), but them’s the breaks. Apparently, I’m not the first to feel this way and others have tried and are trying to do something about it.

What about all that bit mask stuff? Well that’s now pretty simple to sort out as well. One common convention that you need for a bit of additional perspective is to know that “base-16” is often adopted for representing bytes. The reason is that a byte is 8 bits and each 4 bits represents one of 16 possible values. So if you have a base-16 number, bytes can be written with two values. These take the form of something like 4E because you get 0-9 of base-10 that you’re used to and then another 6 values representing 10 through 15 which are written as the characters A through F. It’s much easier to remember bytes as two values than it is to remember 16 bits. I mention all of this because “FF” is identical in value to “11111111” and “F0” is identical to “11110000”.

You don’t see bit masking as much in managed languages, though you might if you get your hands dirty down in the transport layer doing things like socket management. But those coming from a C background are quite familiar with it. If you get a message over the wire, it may come in as a stream of bytes. But when things like network latency and accuracy are issues (or perhaps in embedded environments when space is at a premium) there is a tendency to squeeze every last modicum of value out of the space available. In a web application, representing a number 1 through 10 with data type “int” (which in managed languages will be a 32-bit/4-byte integer) is not really an issue, but in the environments I’ve mentioned, it’s digital blasphemy to waste that space. You only need 4 bits to represent that — the other 28 are wasted.

But the smallest unit typically dealt with in programming convention is the byte. So what to do? Well, enter bit masking. Let’s say I have two speakers on some embedded device that can be set to volume level 1-10 and I want to write a protocol for querying volume level for both. What I would do is define a message that was 1 byte long and have the first 4 bits of the byte represent the volume of one speaker and the second 4 represent the volume of the other. If I wanted just the value of speaker 2, I would want a way to preserve the first 4 bits of the byte I got while clobbering the second 4. Well, enter the bit mask. If I perform the operation int myValue = 0x0F & incomingSpeakerByte, I get what I’m looking for. Why? Well, let’s consider this.

The value 0x0F can be rewritten as 0x00001111 and the value of the speakers is whatever is coming back. Let’s say that it’s set to volume level 9, which can be rewritten as 0x1001. But, let’s say that the other speaker is set to volume level 4, which is 0x0100. That means that both speakers together are represented with the byte 0x01001001. If I set a local variable (where space is not a premium) to the byte I’m getting over the wire, it will be equal to 73, and that is nonsense in our 1-10 volume domain. But the bit mask operation performs the operation 0x00001111 & 0x01001001 to get the final value of 9 (0x00001001) correctly. The reason this works is that the “&” operation aligns the two sequences of bits and performs AND on them one by one, recording the value. This is called “bitwise and.” If you think back to the domination and identity laws from the last post, we’re using domination (x & 0 = 0) to clobber the first 4 bits and identity (x & 1 = x) to preserve the second 4, which are the only four that interest us. (Storing the other byte in a local int is a bit more involved, taking advantage of the aforementioned “bit-shifting,” and beyond the scope of this post). The long and short of it is that when you see bit masking operations performed, you now know what’s going on — the developer is using the same byte to store multiple values and using this technique to extract the ones he or she needs at the moment.

As to thinking in ones and zeroes, you might not be quite there yet, but this is a step in the right direction. It helps you to understand the fundamental building blocks of software based data and how we get from the physical world to the “soft” representation that we deal with. It also hopefully makes some of the weird-symbol bit manipulation tricks seem a little less like black magic. At the very least, I hope that instead of viewing some of this stuff with suspicion or hopelessness, you now have some basic concepts, terms to google and questions to ask.

Further Reading

  1. Wikipedia on bitwise operations
  2. HowStuffWorks on bits and bytes
  3. How many x in a y calculator
  4. Wiki on basic bit manipulation
« Older Posts
Newer Posts »