What is a hack?

Before we move on to even dirtier and more dangerous hacks, lets first define what constitutes a hack. IMO, a programming hack must fulfil the following criteria:

A hack uses a non-sanctioned way of performing a task - it bypasses official and documented APIs. It uses a "clever" and non-intuitive method to perform a task that would otherwise be hard or impossible to do.

Cons:

Usually makes your code harder to maintain

Fragile code, breaks easily when the code it depends upon changes

Hard to port to other platforms

Refactoring, improving the design, is often a better solution

Pros:

No other way to perform a task - the alternative is to rewrite or re-implement OPC (Other People's Code)

Potential for improved performance

Curiosity, gaining a better understanding of the system

Its great fun!

The general advice is to avoid hacks in production code. Hacks can be very useful as short-term solutions during research and development, but is best replaced with refactored code and design before going to production. That said, most real-world software contain the odd hack or two.

Why blogging - a waste of time?

There have been various discussions about the blogging phenomena and some worries that too much sensitive company information could bleed out to the public. Other people worry about the time "wasted" in writing and reading blogs. As Captain Jake put it over in delphi.non-tech:

> Who has the time to write it all?

Good point, actually. I'm geeky enough to (admit that) I use a laptop on the tram back-and-forth to work. That's about 1 hour per day. For some time now, I've spent that tram-time (and some evenings, I must admit) in writing TDM articles and tech editing Xavier's and Jon's books. As Jon was taking a hike in the Grand Canyon (great pictures! - well worth a look), I had some idle time between chapters.

I wasn't up to writing a complete magazine article, so I decided to try and join this blogging craze. This means that as new chapters start dropping into my inbox, the rate of my blog articles will most certainly decline. Still, I like the persistence of a blog and the fact that I can write anything I like (within the limits of law). And the conformity of RSS and Atom feeds means that it is easy for readers to know when there is something new to read, without having to poll a web page manually.

Newsgroups are nice, but the signal-to-noise ratio for interesting information is irritatingly low. It is much more productive to monitor a selected list of relevant and high quality blogs. I was talking to a friend, showing him the hack to gain write access to the Screen.PixelsPerInch property. He just shook his head and said;

"Hack is your middle name".

Thus the idea of writing about some of the hacks I've played with over the years materialized. Hopefully, I'll find the time to write about less suspicious programming techniques and more about higher level topics some time in the future... :-).

Hack #4: Access to protected methods

As we saw earlier, it is possible to gain write access to a read-only property, or rather to its backing field (am talking about native code Delphi here, Win32 and Kylix, - the situation is quite a bit stricter and cleaner in .NET land). You can promote protected properties to public or published to gain access to them, but there is no corresponding promotion syntax for fields or methods. Often you absolutely need to call a protected method of some class - most often method in a VCL component or a 3rd party utility class. This is usually the sign of a badly design class. The author didn't think that application level code would need access to the method. A mock-up example would be:

unit Utility;

interface

type
  TUtility = class
  protected
    procedure Useful;
  public
    constructor Create;
    procedure Boring;
  end; 

We want to call the protected Useful method, but all we got access to is the public Boring method. Typical, right? ;) If you are lucky, you have control over the type of the instance you are using, and can write your own descendent that promotes the method the clean way, like this:

type
  TMyUtility = class(TUtility)
  public
    procedure Useful;
  end;

procedure TMyUtility.Useful;
begin
  inherited Useful;
end; 

Now you can use TMyUtility instead of TUtility and you'll have the Useful method available as a public method. A final twist is to use the same class name, like this:

unit MyUtility;

interface

uses
  Utility;

type
  TUtility = class(Utility.TUtility)
  public
    procedure Useful;
  end; 

Now you only need to add the MyUtility reference in the uses clause after the Utility reference. However, sometimes you cannot control the type of the instance used - it is created inside a body of 3rd party code, for instance.

As many experienced Delphi programmers will know, you can easily gain access to the protected members of a class instance, simply by casting the instance to a descendent class type declared in the same unit as the cast-expression. For instance, to call the protected Useful method of an instance of the TUtility class you can just:

type
  TUtilityEx = class(TUtility)
  end;

procedure Foo(Instance: TUtility);
begin
  // Fails to compile, cannot call protected method
  Instance.Useful;   
  // compiles fine, calls protected method
  TUtilityEx (Instance).Useful; 
end; 

The way the Delphi visibility directives work, all code in the same unit as the declared class will gain full public access to all members. It is a kind of implicit friend concept as known from C++.

Thanks to the ingeniousness of Borland's compiler developers this common hack even works in .NET with Delphi 8 - as long as the caller and method is in the same assembly - it does not work across assemblies, because protected maps to the family-or-assembly method attribute in the CLR.

One very prominent component vendor even routinely hides away truly useful (and sometimes essential) methods of their component and helper classes into the protected section, seemingly only in an attempt to hide complexity from naïve users. Then in all their own units that need access to the methods, they turn around and perform this local-descendent-cast hack and their own classes. Shudder! This hack should only be used to get access to other people's code, and not used between your own units!

Interestingly, the compiler only uses the local hack-class to get access to the protected method. Although the TUtilityEx class is mentioned in the source code, nothing from that class (the VMT, RTTI, classname info etc.) is needed or referenced in the compiled code, so the smart linker removes it from the executable code. Otherwise this hack would be very expensive with regards to code-size.

Thanks again, Danny & co!

Hack #3: Enabling the JIT Optimizer for D8 for .NET

(or the hack that wasn't). You may be getting less than you bargained for in your Delphi 8 compiled assemblies and executables, due to the seemingly hard-coded setting for the Debuggable attribute in the dccil compiler.

.NET and the CLR define a DebuggableAttribute with two boolean constructor parameters and corresponding properties, named IsJITTrackingEnabled and IsJITOptimizerDisabled. When present in an assembly the CLR's JITer will modify its machine code generation strategies to better suit debugging needs.

IsJITTrackingEnabled = True means that the JITer spends time and memory during the just-in-time compilation process to generate tables that makes it easier for a debugger (mapping code-addresses back to source line number information, for instance).

IsJITOptimizerDisabled = True means, well, just that - you'll get easier-to-debug machine code, but the code will perform worse and/or be larger than the JIT optimizer otherwise could produce.

The trouble with D8 is that dccil seems to be hard-coded to always emit the DebuggableAttribute with the isJITOptimizerDisabled parameter = True. If you compile from the IDE, and turn off the integrated debugger and turn off output of the .PDB file from the Linker options, dccil can be persuaded to output DebuggableAttribute(False, True). But no matter how many different compiler options or command line tweaks I’ve tried, I’ve not been able to make the compiler output DebuggableAttribute(False, False) or no attribute at all.

IMO, the setting of the compiler Optimization checkbox should control the isJITOptimizerDisabled parameter, but it currently doesn’t. To see the parameter values of this attribute in your assembly or EXE file, use Lutz’ Reflector instead of ILDAsm. ILDAsm doesn't show the actual value of this attribute, due to the way ILAsm works when applying the attribute (from command line parameters). Reflector gives a better picture.

We need a little piece of test code to see how the settings of this attribute actually influence the runtime behaviour of CPU-intensive code. Thanks to pp-2ch from the newsgroups for the initial version of this mock-up code:

program TestDebuggable;
{$APPTYPE CONSOLE}
function QueryPerformanceCounter(out lpPerformanceCount: Int64): LongBool; external 'kernel32.dll';
var
  StartCount, EndCount: Int64;
  i, j, k: Integer;
begin
  QueryPerformanceCounter(StartCount);
  i := 0;
  while i < 1000000 do
  begin
    k := ((i*2)+2);
    i := k div 2;
    dec(i);
    j := i;
    inc(j);
    i := j;
  end;
  QueryPerformanceCounter(EndCount);
  Writeln(Convert.ToString(EndCount-StartCount));
  Readln;
end.

Compile this from the IDE or command line with default settings, and load the assembly in Reflector. Enable the display of Custom Attributes. Look at the TestDebuggable node and notice the value of the Debuggable attribute in the lower pane. It says Debuggable(true, true). Now run the test project and notice the runtime ticks.

On my machine it clocks in at around 42000 ticks. The initial (but less than ideal) hack-around would be to explicitly include the Debuggable attribute in the project file. DebuggableAttribute is defined in the System.Diagnostics namespace, se we need to add that to the uses clause. And it can only be applied to assemblies and modules, so we must prefix it with an attribute target specifier, like this:

uses
  System.Diagnostics;

[assembly: Debuggable(false, false)]

If you compile and run this from the IDE, make sure you turn off the integrated debugging (Tools | Options | Debugger Options | [ ] Integrated Debugging). Refresh the list of assemblies in Reflector, just press F5 (I love this new feature). Now you will see two Debuggable attributes, and depending on how (un)lucky you are, you will see them in this order: Debuggable(false, false), Debuggable(true, true).

Run the test code again. On my machine, it now clocks in at an impressing 13000, shaving off 70% of the original run time, so I was lucky . The problem is that I have had other instances where the attribute order was reversed, like this: Debuggable(true, true), Debuggable(false, false). DebuggableAttribute does not support multiple instances of itself (AttributeUsage.AllowMultiple = False), but we explicitly add our own in addition to the one always added by the compiler, so we break this rule. Another rule is that the order of multiple attributes is arbitrary, and cannot be relied upon. So sometimes this hack works, sometimes it doesn’t. Back to square one.

If we look closely at the documentation for DebuggableAttribute and specifically for the IsJITOptimizerDisabled parameter, we’ll see that they say:

“The runtime … might disable certain optimizations based on the values contained within this attribute.”

Notice the word “might”. A little more experimentation has shown that (in the current V1.1 CLR) you will in fact get the JIT optimized code we’re after, if you:

• Don’t run from a debugger (disable the IDE debugger or run from the command line)
  


• Use Debuggable(false, true) (turn off the generation of the .PDB file)

So just make sure you compile without enabling the PDB file (Project | Options | Linker | [ ] Generate .PDB debug info file), or from the command-line use the –V- option. Now the compiler will emit Debuggable (False, True). Now Reflector only shows one instance of the Debuggable attribute (assuming you reverted the failed hack above). And the code runs fast from the IDE (with debugging disabled) and from the command line.

Even if you compile with the IDE debugger enabled, the code will still run fast when running standalone. So the hack wasn’t really needed – just use the right options before compiling the release version (turning off the external debug file), and you will get optimized JITed code. Notice that deleting the .PDB file after compiling has no effect. It is the value of the first parameter to DebuggableAttribute that matters, and the same compiler option controls both.

Hack #2: Fixing System.pas line numbers after D7.1 update

There has been some discussion in the newsgroups about the Delphi 7.1 update and the apparent mismatch between the patched System.dcu and the original System.pas source unit. Unfortunately, Borland did not include it in the original D7.1 patch, but I understand they are planning to include the correct version soon

. The problem occurs when you try to step into RTL code that resides in the System unit. For instance, fire up a new Console project (File | New | Other | Console Application - yeah, we D7'ers do envy you D8'ers the File | New | Customize... command) and add the following high-tech code:

program Project1; 
{$APPTYPE CONSOLE} 

begin 
  Writeln('Look - no hands, ma!'); 
end.

Now enable the RTL/VCL units with debug info enabled by selecting Project | Options | Compiler | [X] Use Debug DCUs. Put the cursor on the Writeln line and press F4 to run to cursor (assuming you use the 'correct' keyboard bindings). Then press F7 to step into the code. With the D7.1 update applied you'll now find yourself at the end of the _LStrOfChar procedure inside System.pas. In other words, you're lost.

You'll also see that the blue dots signifying breakpointable code-lines are not aligned correctly with the actual source code. This is a typical sign of a mismatch between the source code and the actually compiled code. So until Borland gets the correct version of System.pas published in an updated update, we are stuck.

Or are we? Hackers like us don't like twiddling our thumbs, declaring defeat. If you look a little more closely at the mis-aligned blue dots and the wrongly positioned cursor in System.pas (it should have been down in Write0LString a few lines below), we can quickly determine how many lines we're off.

To get it right, we have to delete some lines - 8 lines, actually. But what should we delete?. If we scroll back to the top of the System.pas unit, we find some interesting, but less-than-useful comments about the GNU GPL license. As luck should have it, there are 8 lines in there. So, I delete them. And re-run the test application. Lo and behold, we now end up were we expect - in the Write0LString function! Great! A quick testing shows that other System routines now map to the correct line numbers too. Neat!

The conclusions are:

• The original D7.0 System.pas included that extra 8 lines with GNU GPL info
  


• The D7.1 compiled System.dcu was apparently compiled with a version of System.pas without that extra comment header
  


• The D7.1 update did *not* include the updated System.pas
  


• Apparently, no other line-number changing modifications were made to System.pas

And finally: This is only a short-term stop-gap measure until the official update with the correct System.pas is available from Borland. That file probably will contain other changes as well (otherwise System.dcu wouldn't have been updated in the first place).

Update: The System unit was updated due to a 1K buffer-size issue in LoadResString, see BDN article.

Hack #1: Write access to a read-only property

The other day I was faced with the task of making our main application behave better systems with different screen resolutions (or rather pixel density, as in pixels per inch). This is the classic Large Font/Small Font problem, and getting forms and dialogs to scale properly to show readable fonts and text on all displays. There are several things to keep in mind, some of them are covered here - there are more complications due to MDI windows and Form inheritance.

To make my testing easier (and possibly to let the end-user override the default scaling behavior) I decided to let the current screen density (as determined by Screen.PixelsPerInch) be controlled from a setting in the Registry. The built-in Delphi form scaling works reasonably well, and relies on the fact that the form's design-time PixelsPerInch value is different form the run-time Screen.PixelsPerInch value. Now, PixelsPerInch is a read-only, public property of the singleton TScreen class. It is initialized in the TScreen constructor to the number of vertical pixels per inch as returned by the current graphics driver:

  DC := GetDC(0); 
  FPixelsPerInch := GetDeviceCaps(DC, LOGPIXELSY);

For my testing purposes, I wanted to set the value of the PixelsPerInch property without going to the hassle of actually changing my system setup, but to do that I would somehow have to modify the value of the read-only property. Impossible, right? Well, in software, nothing is really impossible. Software is soft, so we can change it :-). Changing the declaration of TScreen to make the property writeable would work, but as Allen has pointed out, making changes in the interface section of RTL and VCL units can have cascading effects, that are often undesirable. Besides, that would not really qualify as a bona-fide hack - it would have been too easy. Nah, lets do something a little more fun ;P. PixelsPerInch is only a public property, so there is no RTTI for it. Lets declare a descendent class, that promotes the property to published:

type 
  TScreenEx = class(TScreen) 
  published 
    property PixelsPerInch; 
  end;

Now, since TScreen indirectly inherits from TPersistent, and TPersistent was compiled in the $M+ mode, published properties in our TScreenEx class will have RTTI generated for them. But PixelsPerInch is still a read-only property - and there is no way our TScreenEx can make it writeable, because the backing field FPixelsPerInch is private, not protected, and so is off-limits for us.

The cunning thing about the RTTI generated for the TScreenEx.PixelsPerInch property, is that it includes enough information about where to find the backing field in the object instance. Open TypInfo.pas and locate the TPropInfo record that describes the RTTI for each property. Included is the GetProc pointer. For backing fields, this contains the offset off the field in the object instance (sans some flag bits). After decoding this offset and adding it to the base address of the object instance, we now can get a pointer to the backing field and thus modify it - voila write-access! Here is the short version:

procedure SetPixelsPerInch(Value: integer); 
begin 
  PInteger(Integer(Screen) + (Integer(GetPropInfo(TScreenEx, 'PixelsPerInch').GetProc) and $00FFFFFF))^ := Value; 
end;

Decoding that is left as an exercisecise for the reader.

First blog post

The blogmania never seems to stop, so now even I have been enticed into trying this new fad... :)