.NET 2.0 for Delphi programmers available now

Jon Shemitz’ classic .NET 2.0 for Delphi Programmers book is now available in hardcover and shipping from Amazon and other online and old-fashioned book shops. I don’t have my own copy yet – if you already have yours, you can be the first to write an Amazon review. Read my review of it here. As I’ve mentioned before there is a sample chapter online at Jon’s site.

Highly recommended!

Simple Interface RTTI

Delphi supports getting RTTI for all interfaces, but it does not include method information for “normal” interfaces.

type
  {$M-}
  IMyMMInterface = interface
    procedure Foo;
  end;  

By using the built-in TypeInfo function on the interface type, we get a pointer to the RTTI structure the compiler has generated for it, a pointer to a TTypeInfo record. This record is defined in the TypInfo unit and looks like this:

  PPTypeInfo = ^PTypeInfo;
  PTypeInfo = ^TTypeInfo;
  TTypeInfo = record
    Kind: TTypeKind;
    Name: ShortString;
   {TypeData: TTypeData}
  end;

Just following the TTypeInfo record in memory is another variable-length variant record that contains different fields and information depending on the value of the Kind field above. The supported type kinds are defined like this:

type
  TTypeKind = (tkUnknown, tkInteger, tkChar, tkEnumeration, tkFloat,
    tkString, tkSet, tkClass, tkMethod, tkWChar, tkLString, tkWString,
    tkVariant, tkArray, tkRecord, tkInterface, tkInt64, tkDynArray);

For interfaces, we’re only interested in the tkInterface kind. The part of the TTypeData record that encodes the RTTI for normal interfaces look like this:

type
  TIntfFlag = (ifHasGuid, ifDispInterface, ifDispatch);
  TIntfFlagsBase = set of TIntfFlag;
  // …
  PTypeData = ^TTypeData;
  TTypeData = packed record
    case TTypeKind of
      // …
      tkInterface: (
        IntfParent : PPTypeInfo; { ancestor }
        IntfFlags : TIntfFlagsBase;
        Guid : TGUID;
        IntfUnit : ShortStringBase;
       {PropData: TPropData});
      // …
  end;

From this we can se that we have access to the following type information for any interface:

• A pointer to the parent interface (IntfParent)
  
  
• Flags indicating if this interface has a GUID, if it is a dispintf interface and if it is a IDispatch interface (IntfFlags)
  
  
• The GUID of the interface (if it has one) (Guid)
  
  
• The unit the interface was declared in (IntfUnit)
  
  
• The number of methods in the interface (PropData.Count)

We can write a simple function that dumps this information for a given interface.

type  
  PExtraInterfaceData = ^TExtraInterfaceData;
  TExtraInterfaceData = packed record
    MethodCount: Word;   { # methods }
  end;  
  
function SkipPackedShortString(Value: PShortstring): pointer;
begin
  Result := Value;
  Inc(PChar(Result), SizeOf(Value^[0]) + Length(Value^));
end;  
  
procedure DumpSimpleInterface(InterfaceTypeInfo: PTypeInfo); 
var
  TypeData: PTypeData;
  ExtraData: PExtraInterfaceData;
  i: integer;
begin
  Assert(Assigned(InterfaceTypeInfo));
  Assert(InterfaceTypeInfo.Kind = tkInterface);
  TypeData := GetTypeData(InterfaceTypeInfo);
  ExtraData := SkipPackedShortString(@TypeData.IntfUnit);
  writeln('unit ', TypeData.IntfUnit, ';');
  writeln('type');
  write('  ', InterfaceTypeInfo.Name, ' = '); 
  if not (ifDispInterface in TypeData.IntfFlags) then
  begin
    write('interface');
    if Assigned(TypeData.IntfParent) then
      write(' (', TypeData.IntfParent^.Name, ')');
    writeln;
  end
  else  
    writeln('dispinterface');
  if ifHasGuid in TypeData.IntfFlags then
    writeln('    [''', GuidToString(TypeData.Guid), ''']');
  for i := 1 to ExtraData.MethodCount do  
    writeln('    procedure UnknownName',i,';');
  writeln('  end;');
  writeln;
end;  

The function expects a pointer to the type information of an interface. It digs out the generated interface RTTI and tries to write a pseudo interface type declaration inside a unit using the information available. It only knows the number of methods in the interface, so it just outputs some dummy names for them.

To test this we can define some plain vanilla ($M-} interfaces and then use the TypeInfo intrinsic function to get a RTTI pointer for each interface to send to the dumping function above.

program TestSimpleInterfaceRTTI;
  
{$APPTYPE CONSOLE}
  
uses
  SysUtils,
  TypInfo;
 
// ... Insert the code above here
 
type
  {$M-}
  IMyInterface = interface
    procedure Foo(A: integer);
    procedure Bar(const B: string);
    procedure Nada(const C: array of integer; D: TObject);
  end;  
  IMyDispatchInterface = interface(IDispatch)
    ['{9BC5459B-6C31-4F5B-B733-DCA8FC8C1345}']
    procedure Foo; dispid 0;
  end;  
  IMyDispInterface = dispinterface
    ['{8574E276-4671-49AC-B775-B299E6EF01C5}']
    procedure Bar;
  end;  
 
begin
  DumpSimpleInterface(TypeInfo(IMyInterface));
  DumpSimpleInterface(TypeInfo(IMyDispatchInterface));
  DumpSimpleInterface(TypeInfo(IMyDispInterface));
  readln;
end.

Running this project produces the following output:

unit TestSimpleInterfaceRTTI;
type
  IMyInterface = interface (IInterface)
    procedure UnknownName1;
    procedure UnknownName2;
    procedure UnknownName3;
  end;

unit TestSimpleInterfaceRTTI;
type
  IMyDispatchInterface = interface (IDispatch)
    ['{9BC5459B-6C31-4F5B-B733-DCA8FC8C1345}']
    procedure UnknownName1;
  end;

unit TestSimpleInterfaceRTTI;
type
  IMyDispInterface = dispinterface
    ['{8574E276-4671-49AC-B775-B299E6EF01C5}']
    procedure UnknownName1;
  end;

We are able to pick up the name of the unit (or program as it is in this case) the interface is declared in, the name of the interface and its parent interface and the GUID if it has one. We can also distinguish dispinterfaces that is used in the implementation of Automation servers (for dual interfaces that inherit from IDispatch).

As you can see we are missing the proper names of the interface methods and we don’t have any information about the parameters, return types or calling conventions. Compiling interfaces in the $M+ mode (or inheriting from IInvokable) changes all that – as we shall see in an upcoming article.

Blog feed: atom2rss converter down

As one of my diligent readers made me aware of, the free service I've been using to convert Blogger's Atom-only based XML feed to the more widespread RSS format has stopped working for some time now. The service I used is documented here:

http://www.2rss.com/software.php?page=atom2rss

When I initially created this blog, I included a link of the following format

http://www.2rss.com/atom2rss.php?atom=http://hallvards.blogspot.com/atom.xml

to an XML icon () that used to be listed under the Syndication header in the sidebar. I've decided to comment out that link for now.

So for those of you who are sourcing the RSS feed instead of the Atom feed (and thus will not be able to read this :)), please switch to the Atom feed instead:

http://hallvards.blogspot.com/atom.xml

Sorry for the inconvenience!

Digging into SOAP and WebSnap

In previous articles on this blog we have dissected the internal workings of published methods, and lamented the (assumed) fact that the RTTI for them does not include signature information. We then developed a dirty (and pretty useless) hack to dig out the parameters of a published method by matching it with the RTTI of an event that references the method at runtime. As I briefly mentioned in my last post, David Glassborow pointed me to the extended RTTI that Delphi 6 and later supports.

In Delphi 6, Borland introduced new web services functionality to be able to write SOAP servers and clients. They also released WebSnap – a framework for writing ASP-like web applications with server-side scripting. Both of these frameworks need the functionality to call arbitrary methods with any number and types of parameters dynamically at runtime.

SOAP and method RTTI for interfaces
For SOAP-based Web services, they use invokable interfaces to let a THTTPSoapPascalInvoker component dynamically call methods in a registered interface. An invokable interface is one that inherits from IInvokable, and IInvokable is compiled using the compiler directive $M+ (see System.pas). The compiler generates full RTTI for all the methods in invokable interfaces – including enough parameter information to call the methods dynamically at runtime.

You can read the juicy technical implementation details by browsing the code in IntfInfo.pas, Invoker.pas and RIO.pas. Note that full source for these were not made available in D6 and D7 (there is a RIO.int in the Doc directory, though). They can be found in the Source\Win32\SOAP directory in D2005 and 2006.

WebSnap and method RTTI for classes
For WebSnap, the “new” $METHODINFO ON compiler directive (which was undocumented in D6 and D7) is used to generate extended RTTI for public and published methods. A class declared in this mode (and all classes that inherit from it) will have RTTI generated for all its public and published methods. In addition to the normal $M+ style published method RTTI with name and address, the extended RTTI for METHODINFO includes the name and type of all parameters, the return type and calling convention. This allows the methods (and published properties) to be accessed from the server-side WebSnap script.

In the WebSnap directory you’ll find the source of the following interesting units ObjAuto, ObjComAuto and WebSnapObjs.

Exploring the fascinating and impressive implementation details in these units is left as an exercise for the reader. We might revisit them for a closer examination in the future.

In coming articles, we’ll look into the details of simple interface RTTI, extended interface RTTI and extended class method RTTI. Stay tuned!

David Glassborow on extended RTTI

It turns out the story (part one, two and three) about getting RTTI for published method parameters isn’t over yet :-).

In Delphi 7, Borland extended the RTTI capabilities in order to support SOAP and WebSnap by introducing the (then undocumented) $METHODINFO compiler directive. We’ll look at this in more detail later, but in the mean time make sure you go to David Glassborow’s blog to read his great posts on Interface RTTI and Class RTTI!

He has even written a DetailedRTTI unit that can return a string representation of a method’s signature – without any event property hacks (but only for classes compiled with $METHODINFO ON). Note that his unit has some interesting record helpers and class helpers (so you need Delphi 2006 to compile it) – read about David’s view on class helpers as a design tool here.

Updated (27. Oct 2007): $METHODINFO was first available in Delphi 7, not Delphi 6.

Published fields details

In the previous article we looked at how published fields are used by the IDE and VCL to make component references easy to use and to find class references from component type strings. Now we want to dig deeper down into the implementation details of published fields. Starting with analyzing the assembly code in TObject.FieldAddress I was able to reconstruct these approximate Pascal structures:

type
  TPublishedField = packed record
    Offset: Integer;
    Filler: word;  // ??
    Name: {packed} Shortstring; // really string[Length(Name)]
  end;
  PPft = ^TPft;
  TPft = packed record
    Count: Word;
    Filler: LongWord; //??
    Fields: array[0..High(Word)-1] of TPublishedField; // really [0..Count-1]
  end;
  
  PVmt = ^TVmt;
  TVmt = packed record
    // ...
    FieldTable        : PPft;
    // ...
  end;

The FieldTable field in the TVmt structure we’re reverse engineering is defined as a PPft, a pointer to a published field table. The Pft starts with a 2-byte count, and then there is four unknown bytes skipped by TObject.FieldAddress and then an array of variable length TPublisedField records. As in other RTTI structures the shortstring fields are packed so that they only take up the enough space to hold a length byte and the name string. The TPublishedField record contains an Offset into the object instance where the field can be found, 2 unknown bytes and the packed shortstring with the name of the field. We’ll figure out the meaning of these unknown fields shortly.

Luckily, the GetFieldClassTable routine in the implementation section of the Classes unit (which we discussed in the last article), documents clearly that the Filler field of the TPft record points a list of class references. With this information we can update our structures.

type
  PClass  = ^TClass;
  PPublishedFieldTypes = ^TPublishedFieldTypes;
  TPublishedFieldTypes = packed record
    TypeCount: word;
    Types: array[0..High(Word)-1] of PClass; // really [0..TypeCount-1]
  end;
  TPft = packed record
    Count: Word;
    FieldTypes: PPublishedFieldTypes;
    Fields: TPublishedFields; // really [0..Count-1]
  end;

Now we have identified the FieldTypes field that points to a record with a TypeCount and an array of class references. Note that the class references have an extra level of indirection. TClass references are already pointers, but the array actually contains pointers to TClass references. The reason for this is to support RTTI info and TClass VMTs that reside in different modules (packages). We see the same indirection by pointer in the TypInfo unit’s use of PTypeInfo pointers, in the implementation of global variables across units and the InstanceSize and Parent (class) fields of the TVmt. The Delphi package support code generated by the linker automatically fixes up these pointers after all static packages has been loaded by the application.

We still have the unknown filler field in the TPublishedField record. When I first started to write test code and dump this field from the RTIT of selected test classes, it looked like a sequential field index as the values started at 0 and increased steadily; 1, 2, 3. But when I added a second published TObject field, the next index was 0. Hmm. Combined with the seemingly missing link to the FieldTypes array I quickly realized that the unknown TPublishedField was an index into the type reference array.

This also confirmed that the FieldTypes array only contains unique class references. If you have 10 published TLabel fields, there will be only 1 TLabel reference in the FieldTypes array. For large forms with many components of the same type, this saves a little space in the TPublishedField record – each type index is only 2 bytes, while a direct TClass reference would take up 4 bytes. More importantly, the FieldTypes array can now be used to quickly translate from a component name string into a class reference, without wasting time scanning though duplicate class references. As we saw in the last article, this is just what the private TReader.FindComponentClass method does.

After digging through and figuring out the meaning of all the fields, we now have the following type declarations.

type
  PClass  = ^TClass;
  PPublishedField = ^TPublishedField;
  TPublishedField = packed record
    Offset: Integer;
    TypeIndex: word;  // Index into the FieldTypes array below
    Name: {packed} Shortstring; // really string[Length(Name)]
  end;
  PPublishedFieldTypes = ^TPublishedFieldTypes;
  TPublishedFieldTypes = packed record
    TypeCount: word;
    Types: array[0..High(Word)-1] of PClass; // really [0..TypeCount-1]
  end;
  TPublishedFields = packed array[0..High(Word)-1] of TPublishedField;
  PPft = ^TPft;
  TPft = packed record
    Count: Word;
    FieldTypes: PPublishedFieldTypes;
    Fields: TPublishedFields; // really [0..Count-1]
  end;

Apart from the FieldTypes array and the TypeIndex field, this looks strikingly similar to the RTTI structures for published methods. To get a kick-start with writing the utility routines to search and iterate the field table structures I simply used the age-old copy-and-paste and search-and-replace trick.

function GetPft(AClass: TClass): PPft;
var
  Vmt: PVmt;
begin
  Vmt := GetVmt(AClass);
  if Assigned(Vmt)
  then Result := Vmt.FieldTable
  else Result := nil;
end;
 
function GetPublishedFieldCount(AClass: TClass): integer;
var
  Pft: PPft;
begin
  Pft := GetPft(AClass);
  if Assigned(Pft)
  then Result := Pft.Count
  else Result := 0;
end;

The cryptically named GetPft function returns a pointer to the published field table given a class reference. It uses the GetVmt function to get a pointer to the “magic” part of the virtual method table (VMT) and then simply return the value of the FieldTable field. The GetPublishedFieldCount function returns the number of published field in a given class reference (not including the fields of parent classes).

The routines to iterate the published fields of a class using both the index-based lookup and a GetFirst/GetNext based iterators also converted cleanly.

function GetNextPublishedField(AClass: TClass;
  PublishedField: PPublishedField): PPublishedField;
begin
  Result := PublishedField;
  if Assigned(Result) then
    Inc(PChar(Result),   SizeOf(Result.Offset)
                       + SizeOf(Result.TypeIndex)
                       + SizeOf(Result.Name[0])
                       + Length(Result.Name));
end;

function GetPublishedField(AClass: TClass; 
  TypeIndex: integer): PPublishedField;
var
  Pft: PPft;
begin
  Pft := GetPft(AClass);
  if Assigned(Pft) and (TypeIndex < Pft.Count) then
  begin
    Result := @Pft.Fields[0];
    while TypeIndex > 0 do
    begin
      Result := GetNextPublishedField(AClass, Result);
      Dec(TypeIndex);
    end;
  end
  else
    Result := nil;
end;

function GetFirstPublishedField(AClass: TClass): PPublishedField;
begin
  Result := GetPublishedField(AClass, 0);
end;

The only real difference here is that the TPublishedField record does not contain a field with the explicit size of the variable sized record (as the case is with TPublishedMethod). Instead we must use the size of the fixed fields plus the length of the name field to move the current pointer to the next record in the array. As before, caller is responsible for calling GetNextPublishedField the correct number of times (using GetPublishedFieldCount).

Then we have the searching routines that find a specific published field given different searching criteria, such as field name, field offset or field address. These use the iteration functions above. If successful they return a pointer to the relevant TPublishedField record inside the RTTI structures, otherwise they return nil.

function FindPublishedFieldByName(AClass: TClass; 
  const AName: ShortString): PPublishedField;
var
  i : integer;
begin
  while Assigned(AClass) do
  begin
    Result := GetFirstPublishedField(AClass);
    for i := 0 to GetPublishedFieldCount(AClass)-1 do
    begin
      // Note: Length(ShortString) expands to efficient inline code
      if (Length(Result.Name) = Length(AName)) and
         (StrLIComp(@Result.Name[1], @AName[1], Length(AName)) = 0) then
        Exit;
      Result := GetNextPublishedField(AClass, Result);
    end;
    AClass := AClass.ClassParent;
  end;
  Result := nil;
end;

function FindPublishedFieldByOffset(AClass: TClass; 
  AOffset: Integer): PPublishedField;
var
  i : integer;
begin
  while Assigned(AClass) do
  begin
    Result := GetFirstPublishedField(AClass);
    for i := 0 to GetPublishedFieldCount(AClass)-1 do
    begin
      if Result.Offset = AOffset then
        Exit;
      Result := GetNextPublishedField(AClass, Result);
    end;
    AClass := AClass.ClassParent;
  end;
  Result := nil;
end;

function FindPublishedFieldByAddr(Instance: TObject; 
  AAddr: Pointer): PPublishedField;
begin
  Result := FindPublishedFieldByOffset(Instance.ClassType, 
    PChar(AAddr) - PChar(Instance));
end;

Working directly with the TPublishedField pointers returned by the three functions above can be a little awkward, so I’ve also written a few wrapper routines that return simple values for the offset, address and name of a field in a given class or object reference.

function FindPublishedFieldOffset(AClass: TClass; 
  const AName: ShortString): integer;
var
  Field: PPublishedField;
begin
  Field := FindPublishedFieldByName(AClass, AName);
  if Assigned(Field)
  then Result := Field.Offset
  else Result := -1;
end;

function FindPublishedFieldAddr(Instance: TObject; 
  const AName: ShortString): PObject;
var
  Offset: integer;
begin
  Offset := FindPublishedFieldOffset(Instance.ClassType, AName);
  if Offset >= 0
  then Result := PObject(PChar(Instance) + Offset)
  else Result := nil;
end;

function FindPublishedFieldName(AClass: TClass; 
  AOffset: integer): Shortstring; overload;
var
  Field: PPublishedField;
begin
  Field := FindPublishedFieldByOffset(AClass, AOffset);
  if Assigned(Field)
  then Result := Field.Name
  else Result := '';
end;

function FindPublishedFieldName(Instance: TObject; 
  AAddr: Pointer): Shortstring; overload;
var
  Field: PPublishedField;
begin
  Field := FindPublishedFieldByAddr(Instance, AAddr);
  if Assigned(Field)
  then Result := Field.Name
  else Result := '';
end;

Finally I wrote some routines to return the type, address and value of a published field, once you have a proper TPublishedField pointer in hand. These are useful when you are writing your own functions that iterate the published fields of a class.

function GetPublishedFieldType(AClass: TClass; Field: PPublishedField): TClass;
var
  Pft: PPft;
begin
  Pft := GetPft(AClass);
  if Assigned(Pft) and Assigned(Field) and (Field.TypeIndex < Pft.FieldTypes.TypeCount)
  then Result := Pft.FieldTypes.Types[Field.TypeIndex]^
  else Result := nil;
end;

function GetPublishedFieldAddr(Instance: TObject; Field: PPublishedField): PObject;
begin
  if Assigned(Field)
  then Result := PObject(PChar(Instance) + Field.Offset)
  else Result := nil;
end;

function GetPublishedFieldValue(Instance: TObject; Field: PPublishedField): TObject;
var
  FieldAddr: PObject;
begin
  FieldAddr := GetPublishedFieldAddr(Instance, Field);
  if Assigned(FieldAddr)
  then Result := FieldAddr^
  else Result := nil;
end;

Phew! Lots of boring plumbing code there. With that under our wings we can write a reverse engineering function that dumps a reconstructed Pascal class declaration containing all the published fields of a class.

procedure DumpPublishedFields(AClass: TClass); overload;
var
  i : integer;
  Count: integer;
  Field: PPublishedField;
  FieldType: TClass;
  ParentClass: string;
begin
  while Assigned(AClass) do
  begin
    Count := GetPublishedFieldCount(AClass);
    if Count > 0 then
    begin
      if AClass.ClassParent <> nil 
      then ParentClass := '('+AClass.ClassParent.ClassName+')'
      else ParentClass := '';
      writeln('type');
      writeln('  ', AClass.ClassName, ' = class', ParentClass);
      writeln('  published');
      Field := GetFirstPublishedField(AClass);
      for i := 0 to Count-1 do
      begin
        FieldType  := GetPublishedFieldType(AClass, Field);
        writeln(Format('    %s: %s; // Offs=%d, Index=%d',
          [Field.Name, FieldType.ClassName, Field.Offset, Field.TypeIndex]));
        Field := GetNextPublishedField(AClass, Field);
      end;
      writeln('  end;');
      writeln;
    end;
    AClass := AClass.ClassParent;
  end;
end;

Just for kicks I wrote a corresponding dumping routine for an object instance that also writes the current value for each field – it is more or less identical to the code above with the addition of a call to GetPublishedFieldValue to get the value of the field in the given instance. Then to test the code, I wrote this:

type
  {$M+}
  TMyClass = class
  published
    A: TObject;
    LongName: TComponent;
    B: TObject;
    C: TList;
    A2: TObject;
    L2ongName: TComponent;
    B2: TObject;
    C2: TList;
  end;

procedure Test;
begin
  DumpPublishedFields(TMyClass);
end;

And the output is:

type
  TMyClass = class(TObject)
  published
    A: TObject; // Offs=4, Index=0
    LongName: TComponent; // Offs=8, Index=1
    B: TObject; // Offs=12, Index=0
    C: TList; // Offs=16, Index=2
    A2: TObject; // Offs=20, Index=0
    L2ongName: TComponent; // Offs=24, Index=1
    B2: TObject; // Offs=28, Index=0
    C2: TList; // Offs=32, Index=2
  end;

Well, that was a lot of fun! :-)

Now we have documented three of the more interesting undocumented VMT fields that point to RTTI information generated by the compiler;

  TVmt = packed record
    // ..
    FieldTable        : PPft;
    MethodTable       : PPmt;
    DynamicTable      : PDmt;
    // ..
  end;

There are still four fields we haven’t looked at yet;

  TVmt = packed record
    // ..
    IntfTable         : Pointer; 
    AutoTable         : Pointer;
    InitTable         : Pointer;
    TypeInfo          : Pointer;
    // ..
  end;

If time and interest permits, we might look at these in upcoming articles.

Acknowledgement. Note that Ray Lischner has documented most of these RTTI structures in his excellent Delphi in a Nutshell book. I'm digging and reverse engineering these structures independently, but it is fun to confirm my findings with what Ray wrote.

Published fields

In our little series about reverse engineering the undocumented fields of the Delphi VMT, we have come to the FieldTable field. This field points to structures that describe the published fields of a class. In Delphi, published fields must be object references and are mainly used by forms and datamodules to store component references in logically named and easy to use fields (the alternative would be to use the Components property array with specific index values and casting).

The Delphi RTL only contains a single exposed method that accesses the field table, TObject.FieldAddress. This method returns the address of a published field given the field name.

function TObject.FieldAddress(const Name: ShortString): Pointer;
asm
        // ...
        MOV     ESI,[EAX].vmtFieldTable
        // ...
end;

This method is used by the component system, in the implementation of the private TComponent.SetReference method, to search the component’s owner for a field that matches the name of the component. If the owned component finds a correctly named published field in its owner, it will assign the field with the component’s reference or nil, depending on the component has just been is added to or removed from the owner.

procedure TComponent.SetReference(Enable: Boolean);
var
  Field: ^TComponent;
begin
  if FOwner <> nil then
  begin
    Field := FOwner.FieldAddress(FName);
    if Field <> nil then
      if Enable then Field^ := Self else Field^ := nil;
  end;
end;

procedure TComponent.InsertComponent(AComponent: TComponent);
begin
  // …
  AComponent.SetReference(True);
  // …
end;

procedure TComponent.RemoveComponent(AComponent: TComponent);
begin
  // …
  AComponent.SetReference(False);
  // …
end;

procedure TComponent.SetName(const NewName: TComponentName);
begin
  // …
    SetReference(False);
    ChangeName(NewName);
    SetReference(True);
  // …
end;

This is how the component fields in your form class automagically get their values. And if you free a component at runtime, the corresponding field is automagically cleared to nil. Pretty neat, huh?! :-).

We’ll look at the exact layout of the field table shortly, but to make FieldAddress work the field table must contain the name of each field and the offset into the object instances it resides. Note that the field table is part of the VMT and thus part of the class, not a specific object instance. This is why the field table cannot contain the actual address of the field, only the offset. The offset must be combined (added to) the object instance address to get the true address of the field at runtime.

There is another, private, routine that also accesses the field table. The Classes unit contains a BASM routine in the implementation section called GetFieldClassTable. This routine accesses a different part of the field table – one that contains class references.

function GetFieldClassTable(AClass: TClass): PFieldClassTable; 
asm
        MOV     EAX,[EAX].vmtFieldTable
        // …
end;

This routine is part of some of the innermost private implementation details of the TReader streaming logic. The nested calls that end up in GetFieldClassTable starts with the public TReader.ReadComponent method and look like this:

TReader.ReadComponent
     CreateComponent (nested routine)
          FindComponentClass
               GetFieldClass
                    GetFieldClassTable
     FindExistingComponent (nested routine)
          FindComponentClass
               GetFieldClass
                    GetFieldClassTable

The FindExistingComponent logic handles visually inherited forms, datamodules and frames. CreateComponent creates a new component read from a DFM stream, given the string of with the component class name. FindComponentClass operates like a local mapping of class name strings to runtime TClass references. Instead of using the heavy-duty global routine GetClass that is used by Delphi at design-time, FindComponentClass first limits its search to the list of unique class types of declared published fields. You see, in addition to the name-offset association, the field table also contains a list of all the unique class types used for published fields in the owner class.

When you design a form, there is a little-known trick to remove the component fields of components you never reference from code. Alternatively, you can simply clear the name field of the component. This will make the component unnamed and the IDE will remove the field declaration for you. These tricks will slightly reduce the size of the DFM and slightly improve the form load performance at runtime.

You have to be careful when performing this trick, however. You must keep at least one published field of each component type on the form, otherwise it will not stream in from the DFM properly – giving you an error message like this:
---------------------------
Debugger Exception Notification
---------------------------
Project richedit.exe raised exception class EClassNotFound with message 'Class TLabel not found'. Process stopped. Use Step or Run to continue.
---------------------------
OK   Help  
---------------------------

Or outside the debugger:
---------------------------
Rich Edit Control Demo
---------------------------
Class TLabel not found.
---------------------------
OK  
---------------------------

You should now see the reason why you get this error. The TReader class uses the list of published field types to convert from a class name string to a proper TComponent class reference. If the class reference is not present in the form class’ field table RTTI, TReader is unable to create the component, and it resorts to raising the EClassNotFound exception you saw above. Note that TReader does fall back to the (potentially) slower GetClass mechanism if the component class reference isn’t found in the field table. This means that an alternative to keeping one published field of each component class, you can call RegisterClass on the component class in an initialization section.

//…
initialization 
  RegisterClass(TLabel);
end.

Then you don’t need any TLabel fields in the form class.

Ok, that should give you some background of why Delphi supports published fields, what kind of RTTI information is stored about them and how the VCL exploits them to perform its design time and DFM streaming magic. The assembly code in TObject.FieldAddress and GetFieldClassTable along with some helpful type declarations inside the Classes unit implementation section give us some helpful clues of how the field table RTTI structures are laid out in memory.

In the next blog post we’ll dive deeper down and write some Pascal data structures and utility methods to find and iterate the published fields and their types. Stay tuned!