Subversion in Delphi's Tools menu

Joe White writes about how to add Tools menu items with Subversion commands
here.

For some reason, the Submit button didn't work on his Comments page (running IE6), so I'll just write my comment here:

Nice, thanks!

I've been meaning to do the same thing for a while, but kept postponing it. Shame about the space-before-macro requirement in the Tools menu - you should probably log it in QC.

I didn't have Ruby installed so I wrote a simple online .Bat file instead:

"c:/program files/tortoisesvn/bin/tortoiseproc.exe" /command:%1 /path:%2 /notempfile

Then I create the Tools items with:
Program: c:\windows\system32\cmd.exe
Parameters: /C C:\SvnPas\Utils\Batch\SvnCmd.Bat diff $EDNAME $SAVEALL

Works fine!

Hack#16: Published field RTTI replacement trick

We're back to fixing the interesting (learning-wise) problem of the flickering TProgressBar on Windows Vista. We have already looked at two relatively dirty and intrusive hacks that either overwrites the TClass reference inside each progress bar instance or overwrites the dynamic method table pointer of the original TProgressBar VMT. There are other related code page overwrite hacks that we may look at in the future (overwriting a virtual method slot, for instance), but this time we will look at a much simpler and (IMO) more elegant solution; tricking the compiler to replace the RTTI it generates for the component field on the for with a fixed version of TProgressBar.

As you know (if you have read the published fields and details articles), the IDE inserts component fields into the unnamed published section of your form class declaration as you drop components and controls on to the form at design time. Since the fields are published, the compiler generates RTTI for them - including the name, type and offset of the field.

For instance, creating a new form and dropping a couple of labels, a button and a progress bar on it, the IDE has generated the following code.

unit Unit1;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, 
  Graphics, Controls, Forms,
  Dialogs, ComCtrls, StdCtrls;

type
  TForm1 = class(TForm)
    Label1: TLabel;
    Label2: TLabel;
    Button1: TButton;
    ProgressBar1: TProgressBar;
  private
    { Private declarations }
  public
    { Public declarations }
  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

end.

The published field RTTI includes the type of each field - encoded as a TClass reference. But how does the compiler know what TClass to use? By using the normal Pascal scoping rules, of course. TLabel and TButton both refers to the classes defined in the StdCtrls unit. For TProgressBar the compiler first checks the StdCtrls unit, but failing to find any TProgressBar class there, it looks in ComCtrls unit and finds it there.

Here lies the clue to our trick - we can simply add another unit to the interface uses clause, one that contains a fixed version of TProgressBar. We just have to make sure this unit is listed after the unit that contains the original TProgressBar (ComCtrls in this case). Let's get to it.

First we write (yet another version of) the fixed TProgressClass and put it inside an aptly named unit.

unit HVProgressBarVistaFix;

interface

uses Messages, ComCtrls;

type
  TProgressBar = class(ComCtrls.TProgressBar)
  private
    procedure WMEraseBkgnd(var Message: TWmEraseBkgnd); 
      message WM_ERASEBKGND;
  end;
  
implementation

procedure TProgressBar.WMEraseBkgnd(var Message: TWmEraseBkgnd); 
begin
  DefaultHandler(Message);
end;  

end.

Notice two things; the class must have the same name as the original class 'TProgressBar' and to avoid a compiler error message like

[Error] HVProgressBarVistaFix.pas(8): Type 'TProgressBar' is not yet completely defined

The class we inherit from has to be explicitly qualified with the unit it comes from 'ComCtrls.TProgressBar'.

With this simple unit in hand, we can fix the progress bar flickering simply by referencing the HVProgressBarVistaFix unit from all forms that uses progress bars (search your code for TProgressBar). Make sure it is listed last in the uses clause of the interface section.

uses
  Windows, Messages, SysUtils, Variants, Classes, 
  Graphics, Controls, Forms,
  Dialogs, ComCtrls, StdCtrls, HVProgressBarVistaFix;

This ensures that the compiler inserts the TClass reference of our fixed version of the TProgressBar class into the published field RTTI tables of the form. When the streaming system reads the .dfm at runtime it uses these published field tables (read the details here) to find the TClass reference (or TComponentClass reference, to be exact) and uses it (and the virtual constructor of TComponent) to create the component. Because we have replaced the ComCtrls.TProgressBar class reference with our fixed version in HVProgressBarVistaFix.TProgressBar, it is our class that will be created.

With this simple kind of hack you can introduce any number of fixes, overriding virtual, dynamic or message methods, for instance. Look ma - no dirty hands! ;).

DN4DP#6: Enumerating collections

This post continues the series of The Delphi Language Chapter teasers from Jon Shemitz’ .NET 2.0 for Delphi Programmers book. Last time we showed how it is now possible to override the meaning of language operators. This time we'll cover the new for in loop and the pattern for introducing enumeration support to your own classes.

Note that I do not get any royalties from the book and I highly recommend that you get your own copy – for instance at Amazon.

"Enumerating collections

To make it easier and more convenient to enumerate over the contents of collections the traditional for statement has been extended into a for in statement. In general the for in syntax is

var
  Element: ElementType;
begin
  for Element in Collection do
    Writeln(Element.Member);
end;

ElementType must be assignment compatible with the type of the actual elements stored inside the collection. The collection must implement the enumerator pattern or be of an array, string or set type. You only have read access to the Element itself, but you can change any properties and fields it references.

All .NET collections and most VCL container classes like TList and TStrings implements the required pattern, so now you can transform old code like

var
  S: string;
  i: integer;
begin
  for i := 0 to MyStrings.Count-1 do
  begin
    S := MyStrings[i];
    writeln(S);
  end;
end;

into the simpler, less error prone, but equivalent

var
  S: string;
begin
  for S in MyStrings do
    writeln(S);
end;

To enable for in for your own collection classes, you need to implement the enumerator pattern. This involves writing a GetEnumerator function that returns an instance (class, record or interface) that implements a Boolean MoveNext function and a Current property. In .NET you can also achieve this by implementing the IEnumerable interface. In Win32 these methods must be public.

type
  TMyObjectsEnumerator = class
  public
    function GetCurrent: integer;
    function MoveNext: Boolean;
    property Current: integer read GetCurrent;
  end;
  TMyObjects = class
  public
    function GetEnumerator: TMyObjectsEnumerator;
  end;

The EnumeratingCollections project demonstrates the differences between the old manual enumeration loops and the new for in loops. It also includes an example of how to write your own classes that support for in enumeration."

DN4DP#5: Redefining the operators

This post continues the series of The Delphi Language Chapter teasers from Jon Shemitz’ .NET 2.0 for Delphi Programmers book. Last time we talked about the new record syntax. This time we will look at the exciting new operator overloading capabilities.

Note that I do not get any royalties from the book and I highly recommend that you get your own copy – for instance at Amazon.

"Redefining the operators

Long-time Delphi programmers have looked at the operator overloading features of other languages (such as C++) with both fear and envy. The ability for user code to redefine the meaning of operators applied to an instance of a specific class is powerful and elegant if used properly, but it can be highly confusing and erroneous if used improperly.

Delphi now supports operator overloading. This is an advanced concept where a class or (more commonly, and the only kind supported in Win32) a record can have a special method called when a specific operator (such as +, -, /, *, div, mod, etc) is applied to an instance of the class or record. To define operator overloads you must define class operator functions with specific names for each operator. See the Delphi language documentation for the full list of class operator names.

In addition to normal operators, implicit and explicit casts (or conversions) can be implemented. The OperatorOverloading project demonstrates the potentially confusing aspects of operator overloading. It defines a TStrangeInt record where the operator semantics have been reversed:

type
  TStrangeInt = record
  public
    Value: Integer;
    class operator Add(const Left, Right: TStrangeInt): 
TStrangeInt; inline;
    class operator Subtract(const Left, Right: TStrangeInt): 
TStrangeInt; inline;
    class operator Multiply(const Left, Right: TStrangeInt): 
TStrangeInt; inline;
    class operator Divide(const Left, Right: TStrangeInt): 
TStrangeInt; inline;
    class operator Implicit(const AValue: Integer): 
TStrangeInt; inline;
    class operator Implicit(const AValue: TStrangeInt): 
Integer; inline;
    class operator Implicit(const AValue: TStrangeInt): 
string; inline;
    class operator Explicit(const AValue: Double): 
TStrangeInt; inline;
  end;

The implementation of a class operator method should create and return a new record or class instance and not modify any of the parameters. Often operator methods are very short and simple and thus perfect inline candidates[1] (inline routines are covered later in this Chapter).

class operator TStrangeInt.Add(const Left, Right: 
  TStrangeInt): TStrangeInt;
begin
  Result.Value := Left.Value - Right.Value;
end;

Invoking the overloaded operators is just a matter of declaring an instance of the record type and using the standard Delphi operators on it.

var
  Strange: TStrangeInt;
  StrangeResult: TStrangeInt;
begin
  Strange := 42;
  StrangeResult := Strange + Strange * 3;
end;

Because of the reversed TStrangeInt implementation, StrangeResult will be 28 instead of the expected 168. See Chapter 7 for more details on operator overloading.

Tip For a more complete example of operator overloading, see the Borland.Vcl.Complex .NET unit (or the Vassbotn.Vcl.Complex unit in the Demos\DelphiWin32\VCLWin32\ComplexNumbers\ folder for the corresponding Win32 unit).

---

[1] Marking overloaded operators with inline is purely optional, of course. Currently (Delphi 2006) it seems like the compiler manages to inline all simple operators except the Implicit and Explicit conversion operators."

Hack#15: Overriding message and dynamic methods at run-time

Last time we looked at a way of completely changing the class of a running object instance. As we discussed there, that hacking technique had a number of problems. But there are many ways to skin a cat (sorry, cat lovers!), and in the case of trying to fix the TProgressBar flickering issue on Windows Vista without changing the interface declaration of TProgressBar there are at least three other possible solutions.

One of these solution is to patch the existing TProgressBar VMT at runtime. We know from earlier articles that the VMT contains a load of information about a class - including an array of all virtual method implementation (the actual VMT - virtual method table) and a sparse array of all overridden and introduced dynamic and message methods (known as the DMT or dynamic method table). In this article we will look at how we can patch an existing class VMT and replace the DMT with one that fits our need.

Note that the compiler stores the class VMT tables in write-protected code pages (even though they do not strictly contain executable code). This means that if we naively try to overwrite parts of the VMT we will get access violations triggered by the hardware memory protection system. The clean way of writing self-modifying code is to use the VirtualProtect API to change the protection of the memory pages to allow writing before performing the patch, then restoring the original protection when we're done. Here is a routine that will safely overwrite a 4-byte DWORD in the code segment:

procedure PatchCodeDWORD(Code: PDWORD; Value: DWORD);
// Self-modifying code - change one DWORD in the code segment
var
  RestoreProtection, Ignore: DWORD;
begin
  if VirtualProtect(Code, SizeOf(Code^), PAGE_EXECUTE_READWRITE,
    RestoreProtection) then
  begin
    Code^ := Value;
    VirtualProtect(Code, SizeOf(Code^), RestoreProtection, Ignore);
    FlushInstructionCache(GetCurrentProcess, Code, SizeOf(Code^));
  end;
end;

To be on the safe side we follow Microsoft's recommended practice of flushing the instruction cache after performing the patch by calling FlushInstructionCache. This is to avoid nasty things from happening if the patched code has already been pre-fetched by (one of) the CPU.

Now we just have to figure out where to patch and what to patch it with. We have covered the details of how dynamic and message methods work in earlier posts. Looking at the source of TProgressBar and confirmed by peeking with the debugger, there are no dynamic or message method overrides at all in this class. This is good news because it means that the DynamicTable pointer in the VMT is nil and this makes it much easier to perform the patch. So now we know where to patch - the DMT slot in the VMT of TProgressBar. To keep things simple in the initial version we use the vmtDynamicTable constant exported by the System unit.

{ Virtual method table entries }
const
  // ...
  vmtDynamicTable      = -48;

Given a TProgressBar class reference we can calculate the address of the DMT slot in the VMT by using this code:

var
  OriginalDmt: PDWORD;
begin
  OriginalDmt := PDWORD(TProgressBar);
  Inc(OriginalDmt, vmtDynamicTable div SizeOf(DWORD));
  if OriginalDmt^ = 0 then // Assert that there is no existing Dmt

Now we have to find what to replace the nil DMT pointer with. We could try to build a DMT table by hand by using the tables we documented here, but it would be wasted effort. The compiler is much better than us building VMTs and DMTs, so we can just use the TProgressBarVistaFix from the previous post.

type
  TProgressBarVistaFix = class(TProgressBar)
  private
    procedure WMEraseBkgnd(var Message: TWmEraseBkgnd); message
WM_ERASEBKGND;
  end;

procedure TProgressBarVistaFix.WMEraseBkgnd(var Message: TWmEraseBkgnd);
begin
  DefaultHandler(Message);
end;

We can get to the DMT pointer for this new class in the same way and use it to overwrite the (previously nil) DMT pointer in the TProgressBar class. This is my initial patching routine:

procedure PatchTProgressBarDMT;
var
  OriginalDmt, NewDmt: PDWORD;
begin
  OriginalDmt := PDWORD(TProgressBar);
  Inc(OriginalDmt, vmtDynamicTable div SizeOf(DWORD));
  if OriginalDmt^ = 0 then // Assert that there is no existing Dmt
  begin
    NewDmt := PDWORD(TProgressBarVistaFix);
    Inc(NewDmt, vmtDynamicTable div SizeOf(DWORD));
    PatchCodeDWORD(OriginalDmt, NewDmt^);
  end;
end;

Testing this code shows that it works and that the WM_ERASEBKFND handler is called and the flicker is gone on Vista. However, I'm not too satisfied with the code, so I refactor it into a separate general unit - this time using the HVVMT unit and the PVmt record pointer.

unit HVPatching;

interface

uses
  Windows;

procedure PatchCodeDWORD(Code: PDWORD; Value: DWORD);

procedure ReplaceClassDmt(TargetClass, SourceClass: TClass);

implementation

uses
  HVVMT;

procedure PatchCodeDWORD(Code: PDWORD; Value: DWORD);
// Self-modifying code - change one DWORD in the code segment
var
  RestoreProtection, Ignore: DWORD;
begin
  if VirtualProtect(Code, SizeOf(Code^), PAGE_EXECUTE_READWRITE,
    RestoreProtection) then
  begin
    Code^ := Value;
    VirtualProtect(Code, SizeOf(Code^), RestoreProtection, Ignore);
    FlushInstructionCache(GetCurrentProcess, Code, SizeOf(Code^));
  end;
end;

procedure ReplaceClassDmt(TargetClass, SourceClass: TClass);
begin
  Assert(Assigned(TargetClass) and Assigned(SourceClass));
  PatchCodeDWORD(@GetVmt(TargetClass).DynamicTable, 
    DWORD(GetVmt(SourceClass).DynamicTable));
end;

end.

Then we can use it like this:

initialization
  ReplaceClassDmt(TProgressBar, TProgressBarVistaFix);
end.

In this case we were lucky. In other cases the patched class may already have one or more dynamic or message methods - so that there will already be a DMT pointer in the VMT. In this case we must play the compiler and manually allocate a new DMT with one extra entry, copy the existing DMT indicies and method code pointers and finally patch in the new dynamic method index or message-id and the address of the hook routine. This is quite a bit more complex, but if there is interest in digging further down we might revisit this later.

Hack#14: Changing the class of an object at run-time

Sometimes (such as when you for one reason or another need to stay backwards compatible with binary dcus) you may have to employ a hack or two. One such hack is to change the actual run-time class of an object instance. You might need to do this to override a virtual, dynamic or message method, for instance.

Vista and ProgressBar flickering

One such case; for some reason (most probably a bug fix) Microsoft changed the behavior of the ProgressBar control in Windows Vista so that it now sends a WM_ERASEBKGND message every time the progress changes and it needs to redraw itself. The TProgressBar component that wraps the native control does not handle WM_ERASEBKGND - nether as an explicit message override or in the general WndProc method (in fact it doesn't override WndProc). The net result is that each time the progress bar's Position value changes noticeable flicker can be observed as the control first erases the background completely and then draws the progress bar in the correct style. Jordan Russell reported this issue in Quality Central #38178.

Since Delphi 2007 was a release to target Windows Vista issues like this, it would have been simple to add a WM_ERASEBKGND override that skips the default TWinControl logic of FillRect'ing the client area.

type
  TProgressBar = class(TWinControl)
  private
    procedure WMEraseBkgnd(var Message: TWmEraseBkgnd); 
  message WM_ERASEBKGND;
  end;

procedure TProgressBar.WMEraseBkgnd(var Message: TWmEraseBkgnd);
begin
  DefaultHandler(Message);
end;

The trouble is of course that Delphi 2007 was to be non-breaking release, so no interface section changes of existing classes was possible. But there are at least three different ways of hacking a solution anyway. In this article we will look at the simplest and probably least robust solution, changing the class of all TProgressBar instances at runtime.

Changing the class

On the face of it, you might think it is impossible to change the class of an existing object instance. The class of an instance is determined by two things; the class type T that the object reference was declared as at compile time and the class type D that was used to create the object instance and assign to the object reference. D must be assignment compatible to T and thus must be T or derived from D.

type
  T = class
  end;
  D = class(T)
  end;

procedure Foo;
var
  Ref: T;
begin
  Ref := D.Create;
end;

While you cannot change the declared type of the object reference, you can change the runtime type of an object instance. Why is this possible? Well, the runtime type of an object instance is stored in an implicit field of the instance memory - the first 4 bytes of an instance always contain the TClass reference (implemented as a pointer to the class' VMT) of the class that was used to create the object. This reserved field is initialized in the InitInstance class method defined in TObject.

class function TObject.InitInstance(Instance: Pointer): TObject;
begin
  FillChar(Instance^, InstanceSize, 0);
  PInteger(Instance)^ := Integer(Self);

This method performs other tasks as well (such as initializing all interface method table fields), but what interests us is that upon entry the Instance parameter points to a raw uninitialized block of memory (you can see that NewInstance calls GetMem and then InitInstance). This block is first cleared to all zeros by calling FillChar (this is what ensures that all object instance fields are 0, nil, false etc) and then overwrites the first 4 bytes with the TClass reference (which is available in the implicit Self parameter for non-static class methods).

Phew! Now that we know that the de-facto runtime class of an object instance is effectively stored as a field with a known offset (0), it is actually amazingly easy to change the class - we can just overwrite the TClass reference there with another TClass value. To ensure that things don't crash (the compiler makes assumptions about field offsets, virtual method indexes and so on based on the declared type of an object reference) we have to be careful to only overwrite the VMT slot with a class that derives from the current one. And since the object instance has already been allocated with a fixed size, we should not add any instance fields to the derived class.

Let's see how we can use this technique in the TProgressBar case. First lets declare and implement a TProgressBar descendant that fixes the Vista issue.

type
  TProgressBarVistaFix = class(TProgressBar)
  private
    procedure WMEraseBkgnd(var Message: TWmEraseBkgnd); 
  message WM_ERASEBKGND;
  end;

procedure TProgressBarVistaFix.WMEraseBkgnd(var 
  Message: TWmEraseBkgnd);
begin
  DefaultHandler(Message);
end;

It is basically identical to the interface-breaking, clean-solution code above - we only had to change the name of the class and declare that it inherits from TProgressBar. Then we have to write the code that takes an existing TProgressBar instance and changes its runtime class to TProgressBarVistaFix.

procedure PatchProgressBar(ProgressBar: TProgressBar);
type
  PClass = ^TClass;
begin
  if ProgressBar.ClassType = TProgressBar then
    PClass(ProgressBar)^ := TProgressBarVistaFix;
end;

Note that it first checks to see that the actual class type is exactly TProgressBar and not some derived type - that would mess things up because TProgressBarVistaFix derives from TProgressBar.

One of the problems with this hack technique is that you need to explicitly patch each progress bar instance. You could do this in the FormCreate of all forms containing one or more progress bars, for instance. Or even in an button's OnClick event handler as I've done in a simple demo app.

procedure TForm5.PatchBtnClick(Sender: TObject);
begin
  PatchProgressBar(ProgressBar2);
end;

If we were CodeGear (or if we feel adventurous) we could apply the patching to a single point in the implementation of the TProgressBar constructor.

constructor TProgressBar.Create(AOwner: TComponent);
begin
  PatchProgressBar(Self);
  // ...
end;

This hack can be useful when you need to quickly change the runtime class of an object on an ad-hoc manner. It has a number of disadvantages, however:

• It requires patching each object instance explicitly
  
• It changes the class of the instance. In our case ClassName will return 'TProgressBarVistaFix', for instance. One workaround for this would be to move the TProgressBarVistaFix class to a separate VistaFixes unit and name it TProgressBar. Still the RTTI and class reference will be different than the original class. 
  
• It doesn't fix the problem for descendants of the patched class. In the TProgressBar case there are probably not so many of those around..?
  
• To ensure that all object instances are patched, you need to have access to the source of and change the implementation of the original class' constructor.
  
• The compiler creates a new VMT for the hack class, so the code size increases accordingly (yeah, who cares?, right).

The benefits of this hack is:

• It is very simple in the sense that it does not require writing into any protected code pages (which would require the use of VirtualProtect or WriteProcessMemory).
  
• It is easy to extend it to override any number of virtual, message and dynamic methods. It can also be used to promote properties to published so that RTTI is generated for them. Just update the hack class and rely on the compiler to build the proper VMT information for you.
  
• It is flexible in that It is possible to patch only selected object instances instead of all instances of a class

Both this hack and the other variants that we will examine in upcoming blog posts have certain deficiencies that is hard to overcome due to the nature of the VMT and compiler. The hacks work well for leaf classes that do not have any descendants, but if you use them to override a virtual method in a non-leaf class (i.e. one that has descendants), and that virtual method is also overridden in one of the descendant classes, and the descendant method calls back to the inherited method, things will not work as if the method had been overridden in the clean way. Clear as mud? We'll revisit this topic later.

Conclusion

The hack presented here of changing the class of an object instance at runtime can be useful in a limited set of cases. Only use it for leaf classes and when replacing the class VMT with a new one doesn't introduce issues (such as ClassName changing). For most other cases, other hacking techniques would be better. We'll look at a couple of these in upcoming articles. Stay tuned! ;)

[Updated: reduced extensive "boldness" of the text ;)]

Delphi 2007 ESD arrived - one gotcha

We ordered Delphi 2007 ESD from the local Norwegian CodeGear distributor alfacode.no a few days ago and earlier today the email with download link and licence keys arrived (the DVD will arrive in a couple of weeks, they say).

I downloaded the installer stub that then downloads and installs the required binaries - it all worked very smoothly. After the install was done Delphi 2007 ran fine. But I was a little worried for a while - because when I tried to start BDS 2006, I got an error dialog with the following unnerving message:

---------------------------
bds.exe - Entry Point Not Found
---------------------------
The procedure entry point @Uxtheme@BufferedPaintSetAlpha$qqruip11Types@TRectuc could not be located in the dynamic link library rtl100.bpl.
---------------------------
OK
---------------------------

After scratching my head for a while, I searched by harddisk for all rtl*.bpl files. Both Delphi 2007 and BDS 2006 use a binary called rtl100.bpl - and normally it resides in the Windows\System32 directory. The Delphi 2007 installer updates this (and most other VCL bpls) when it installs. The problem is that I had a second copy of this file in the BDS 2006 bin directory (called C:\Delphi2006\bin on my system). This was entirely my own fault because I had copied it there and patched it using Peter Vones tool to hack away startup time as I've blogged about before.

The reason this happens is that Delphi 2007 installer upgrades all the *100.bpls in the System32 directory, but BDS 2006 tries to load the old rtl100.bpl from the bds.exe startup directory that is not compatible with the new vcl100.bpls now residing in System32. The solution was simply to delete or rename the rtl100.bpl in the Delphi2006\bin directory.

Lesson learned: it is dangerous to live on the edge - and you better know (and remember) what you are doing.... ;).