DN4DP#3: Nesting habits

This post continues the series of The Delphi Language Chapter teasers from Jon Shemitz’ .NET 2.0 for Delphi Programmers book. Last time we learned about the new class member visibility specifiers that are available, abstract classes and final methods. This time we will look at the syntax and semantics of nested classes.

Note that I do not get any royalties from the book and I highly recommend that you get your own copy – for instance at Amazon.

"Nesting habits

The compiler now allows you to declare nested types (including classes) and constants inside another class or record declaration. The implementation of a nested class method must have its name preceded by both the outer and inner class names, like this

type
  TOuter = class
  public
    const 
      MeaningOfLife = 42;
    type
      TResult = integer;
      TInner = class
      public
        function  Method: TResult;
      end;
  end;
 
implementation
 
function TOuter.TInner.Method;
begin
  // or just MeaningOfLife 
  Result := TOuter.MeaningOfLife; 
end;
 
procedure Test;
var
  Inner: TOuter.TInner;
  Result: TOuter.TResult;
begin
  Inner := TOuter.TInner.Create;
  Result := Inner.Method;
end;

Note that a nested class can reference all types and constants nested within its parent class – the TOuter prefix in the above example is optional. Any code outside of a class must use an explicit type prefix – for example, the external test code in the Chapter10\NestingHabits project must use the TOuter prefix to get at the types and constants declared in the TOuter class.

Remember, while inner classes can see all types declared in their outer class(es), they have no special access to their outer class’ fields, methods, or properties. (Similarly, outer classes cannot see private or protected members of their inner types). The main purpose of nested classes is to reduce namespace clutter by keeping (often private) helper classes within the public classes that use them. This also makes it easier to hide implementation details. (See Chapter 2’s Nested Classes section.)

Tip   Delphi’s global level constants are not CLS-compliant (or rather, the name of the class they end up in is an implementation detail generated by the compiler, and should not be relied upon), so you should declare constants you want to export to other languages inside a class or record declaration. The same advice goes for global routines and variables - declare them as static methods and fields."

Borland and CodeGear

As most of you will already know, Borland has now moved its developer tools group into a separate company called CodeGear. CodeGear, which will still be wholly owned by Borland, will continue to develop, sell and support developer tools like Delphi, C++Builder, JBuilder and Interbase. There are also indications that they are moving into the webspace with support for languages such as PHP, Pyhon, Ruby and Ajax technology.

While some people have been waiting anxiously for a complete separation from Borland in a operation to sell the IDE group to a separate entity, I think that the current situation is better for the stability and long-term viability of CodeGear in general and the future of Delphi in particular.

The CodeGear management will be in a position to grow the IDE business and pour the revenue stream back into the development of new versions and products. I think this is good news, but as always we should keep our heads calm and keep our options open ;).

Good thing we called our user group Oslo Delphi Club and not NoBUG (Norwegian Borland User Group) or something similar ;)).

Hack#13: Access globals faster ($ImportedData)

There is an aura of magic around Delphi packages. Packages allow you to share Delphi code at a much higher level than is possible with plain old DLLs. Writing a DLL-based library API involves writing flat non-OOP global routines and avoiding any data types more advanced than integers, doubles, static arrays, PChar and records. You cannot share or exchange classes, objects, global variables or let alone strings (unless the client is a Delphi app and both the DLL and client use ShareMem).

Delphi Package Magic
Enter the magic of Delphi packages. Now you no longer have to worry about what you can share between module boundaries, because you can share everything! Packages is just a way of dividing up the logical code into physical deployment modules. Everything just works - somehow. The somehow is the (mainly undocumented) magic.

We're going to look at a small part of that magic - how global variables can be accessed across the application/package boundary. Lets start with the simplest possible case first .

Lets say we have a Unit1 that defines a GlobalVar: integer and some code in the same unit that sets it. Here is the generated assembly and machine code.

Unit1.GlobalVar := 13;
004081B1 C705989240000D00 mov [GlobalVar],$0000000d

That is a single assembly instruction with no indirections or pointer access going on. Notice that both the address of the global variable and the immediate value 13 (encoded as a 16-bit shortint, $000D) is encoded directly inside the machine code instruction. When I evaluate the address of the global variable in the Ctrl+F7 evaluator, @GlobalVar, I get $409298. This is the big-endian value I have highlighted in yellow above.

Then let's look at the package magic situation. If we have a PackageA containing a Unit1 that defines a GlobalVar: integer and a DemoApplication that links to PackageA and with code that access the GlobalVar, the compiler will compile the code into something like this:

Unit1.GlobalVar := 42;
00403389 A1A4404000   mov eax,[$004040a4]
0040338E C7002A000000 mov [eax],$0000002a

Notice the indirection there? Global variables are accessed through a pointer that is fixed up to point to the right address inside the package when the package is loaded. That is cool and a good thing. The global variable itself resides inside the package/DLL module, while the application has a magic, "invisible" global pointer variable that points to the actual package variable. This pointer is fixed up during the magic static loading of packages (it's value is probably set to the result of a GetProcAddress API call).

Global variables in standalone apps
Now lets step back a little and look at another common situation. We have looked at the intra-unit situation and the inter-module situation - the two most extreme conditions. But what happens when you access a global variable between units inside the same application? Surely, then the indirection should not be needed? Lets test it.

We write a simple console application consisting of two units, Unit1 and Unit2, and the main program.

unit Unit1;

interface

var
  GlobalVar: integer = 42;

procedure SetGlobalLocally;

implementation

procedure SetGlobalLocally;
begin
  asm int 3 end; // will break here, look in CPU view
  Unit1.GlobalVar := 13;
end;  

end.

unit Unit2;
  
interface

procedure SetGlobalIndirect;

implementation

uses Unit1;

procedure SetGlobalIndirect;
begin
  asm int 3 end; // will break here, look in CPU view
  Unit1.GlobalVar := 42;
end;  

end.

program TestImportedData;
{$APPTYPE CONSOLE}
uses
  Unit1 in 'Unit1.pas',
  Unit2 in 'Unit2.pas';
begin
  SetGlobalLocally;
  SetGlobalInDirect;
end.

Notice the cute little hand-coded breakpoints? Debugger breakpoints are actually implemented by overwriting a byte of your code with the software breakpoint instruction (int 3) which is encoded in a machine code byte $CC. When you run this program with debugging enabled from the IDE, it will break on the two int 3 instructions. Neat, huh? ;)

The first breakpoint is inside Unit1 just above the code that modifies the global variable. This is the case we showed first above. If you run it and follow the instructions to look at the code in the CPU view, you will see that there is no indirection.

Unit1.GlobalVar := 13;
004081B1 C705989240000D00 mov [GlobalVar],$0000000d

The second breakpoint is in Unit2 where we are modifying the global variable from an "external" unit (from a different unit than were the global was declared). Notice that we are still in a single EXE file here, so there should be no need for indirection, as it was in the package case. Run the code again and when it hits the second breakpoint, look at the machine code in the CPU again.

Unit1.GlobalVar := 42;
00403389 A1A4404000   mov eax,[$004040a4]
0040338E C7002A000000 mov [eax],$0000002a

Looks like we still have indirection here...! What is going on? The reason for this (small) inefficiency is that a unit can be moved into or out of a package without being recompiled. So the compiler has to be conservative and generate the unit's code to support the "worst" case that it will be exported for use from a package. The important thing to learn from this is:

By default, all cross-unit global variable access is performed indirectly via a pointer

The implication is that if you have performance critical code that access a global variable, you should cache it in a local variable inside the loop. You can only do this if you do not need to "see" updates to the global variable while the loop is running (think multithreaded code).

For standalone applications that do not use packages at all, it is a bit irritating to know that the compiler generates sub-optimal code for global variable access. While both the performance and size overhead of the indirection should be negligible in most situations, it would still be nice to force the compiler to get rid of this indirection.

$ImportedData Off
And as it happens there is a compiler directive that does exactly this. Enter the $ImportedData (aka $G) directive. This is what the Delphi help file has to say:

"Type Switch
Syntax {$G+} or {$G-}
{$IMPORTEDDATA ON} or {$IMPORTEDDATA OFF}
Default {$G+} {$IMPORTEDDATA ON}
Scope Local
Remarks

The {$G-} directive disables creation of imported data references. Using {$G-} increases memory-access efficiency, but prevents a packaged unit where it occurs from referencing variables in other packages."

By default the setting is {$ImportedData On} - this enables the pointer indirection in code that access unit-external global variables. By using the {$ImportedData On} directive in a unit, you turn off this indirection, resulting in tighter faster code for global variable access. Notice that you have to use it in every unit referencing global variables, not just in the unit declaring it.

Lets add a third unit to our test bench program.

unit Unit3;
 
interface

procedure SetGlobalDirect;

implementation

uses Unit1;

{$IMPORTEDDATA OFF}

procedure SetGlobalDirect;
begin
  asm int 3 end; // will break here, look in CPU view
  Unit1.GlobalVar := 42; 
end;

end.

The code here is identical to the code in Unit2, but now we have included the $ImportedData Off directive to tell the compiler generate optimized code for globals. Run the program again and when it stops at the breakpoint in Unit3, look in the CPU view.

Unit1.GlobalVar := 42; 
004033D1 C705A04040002A00 mov [GlobalVar],$0000002a

Hurrah! We got rid of the indirection.

The {$ImportedData Off} compiler directive forces the compiler to generate efficient global variable access

This works fine in Delphi 6 and 7. As it happens it no longer works in Delphi 2006 (and probably not 2005). Looks like a bug has sneaked into the compiler here. The directive is effectively ignored, still producing the indirection as we saw for Unit2 above. Hopefully Borland/DevCo will be able to fix this little issue in a future version of the compiler[1].

IDE Compiler Options
Knowing that $ImportedData Off generates better code for global variable access in a standalone application, we would of course like to set it for all the units in the application as a whole. Unfortunately that is not so easy to do. You see the Project | Options | Compiler Options dialog page does not give access to this option... :(.

Because of this issue you have to use one of the following solutions:

• Manually insert {$IMPORTEDDATA OFF} at the top of all units in the project
  
• Manually insert an {$ MyAppSettings.Inc} file in all units. The .inc file would contain settings such as {$IMPORTEDDATA OFF}
  
• Compile using Dcc32.exe setting the option from the command-line

In other words it is a bit awkward to enable this for a given application.

It is hard to set {$ImportedData Off} from the IDE.

I hope that they will be able to make this option available from the compiler options in the IDE in a future version. In fact, I went ahead and reported this as a enhancement request in QC - (please vote for it!):

"Can we please have the $G switch (AKA $IMPORTEDDATA) promoted to appear in the Project Options | Compiler dialog? It makes it easier to switch it off globally for those of us who never use run-time packages."

While it can be argued that introducing this IDE option runs the danger of users producing unusable packages, I don't think this is reason to deny the large number of people writing standalone applications easy access to this option.

The solution could be to only enable this option for projects that are compiled with run-time packages turned off (on the Project Options, Packages page).

Still, if a unit is compiled with IMPORTEDDATA OFF and then moved to a package, it has to be recompiled (i.e. the package project must be rebuilt).

To get around this problem you could:

a) Have the compiler detect this case and recompile all units with the wrong IMPROTEDDATA setting

b) Document the issue and require the user to rebuild himself

c) Combined with b) make the option less obvious and make the IDE compiler options future proof by adding a free-text field for additional compiler options. The user must manually type IMPORTEDDATA OFF in this field, supposedly this makes him qualified to know what he is doing.

The following Dcc32.exe compiler options are currently unavailable from the IDE Compiler Options:

G+ Use imported data references
M- Runtime type info
W- Generate stack frames
Z1 Minimum size of enum types"

If you think this is worthwhile to implement, please vote on QC#34650 here!

Score: Delphi 7 vs. Delphi 2006: 1-1 ;)

[1] I found that this issue has indeed been discovered independently and reported by Frederic Vanmol in Quality Central:

Report No: 33464 Status: Open
{$IMPORTEDDATA OFF} does not seem to work
http://qc.borland.com/wc/qcmain.aspx?d=33464

PS. Just as I had finished this piece, I started experimenting some more...;) In Delphi 7 it is actually possible (but still awkward) to compile an application with $ImportedData Off for all units without messing with each unit or the command line compiler.

You only need to

• close the project
  
• manually edit the project's .dof file
  
• change "-G+" to "-G-"
  
• save and close the .dof file
  
• reopen the project
  
• rebuild the application

If you compile from the command line you can update the .cfg file in a similar manner.

It is probably possible to do the same in Delphi 2006 by manually editing the XML in the .bdsproj file, changing

<Compiler Name="G">1</Compiler>

to

<Compiler Name="G">0</Compiler>

But it is a little hard to check, because of the Delphi 2006 bug of ignoring the directive. In both cases, the manually set compiler options are preserved even when changing compiler options from the IDE.

Hack#12: Create smaller .EXE files ($SetPEFlags)

No, this post is not about so-called EXE-compressors - I don't believe in using them. And it is not a pure hack in the sense that we're breaking any rules - its just about documenting an undocumented and unknown feature of the Delphi 2006 Win32 compiler (it is not implemented in D7 - I don't know about D2005 yet as I don't have it installed on this laptop anymore).

Background info
When you compile a DLL (or a package which is a Delphi-specific DLL in disguise), the linker includes what is known as a relocation table. This table includes information about what addresses must be fixed up by the OS loader in the (likely) event that the DLL must be loaded at a different address than its intended-at-compile/link-time base address. You see all DLLs come with a base address that is the "ideal" loading address of that module. The OS will try to load the DLL at this address to avoid the overhead of runtime rebasing (patching the in-memory pages of the DLL forces it to be paged in from disk and prevents cross-process sharing of the DLL pages). That's why you should set the Image base option in the Linker page of the project options of DLL and package projects. The default Image base that Delphi uses is $00400000 for both applications, DLLs and packages - and thus by default all DLLs and packages will need to be rebased - as that address is reserved for the process' EXE file.

The implication is that an EXE file will always be loaded at the fixed virtual address $00400000 and that it will never need to be rebased. Alas, it doesn't really need its relocation table and we can safely remove it, shrinking the size of the .EXE without affecting its behavior or performance. It is basically a free lunch!

Up until now Delphi users have had to resort to external stripping tools like Jordan Russell's excellent StripReloc tool. It basically takes the name of an .EXE file as a parameter and removes the relocation table from it. I normally don't bother to strip the reloc table during development, but I do run Jordan's tool before deploying our application. This strips a hefty 350 kB from our 7 MB EXE file - a nice, free 5% reduction right there.

As far as I know, most Microsoft tools produces reloc-free EXE files, while Delphi always includes it for both DLLs and EXEs. Up until now. Delphi 2006 actually has a unknown, undocumented compiler directive feature that allows you to turn off the relocation table for EXE files.

To test it out I first opened the ResXplor project from the Demos\DelphiWin32\VCLWin32\ResXplor folder (a pretty neat application, btw). I compiled it as-is. Project | Information told me that the size of the EXE file was 614400 bytes (and Windows Explorer agreed). Then I added the following line to the top of the project file:

{$SetPEFlags 1}

(We'll look at the source of the mysterious value 1 in a moment).

Then I recompiled the project (there is no need to rebuild it, as the unit files does not need to be recompiled - only the linker must be reinvoked). And the size of the EXE file had shrunk to 577536 bytes - that is a reduction of 36864 bytes or 6 %. Not bad for a one-liner, eh? ;).

The Delphi 2006 help file has the following to say about the $SetPEFlags compiler directive:

"Type: Flag
Syntax: {$SetPEFlags <integer expression>} {$SetPEOptFlags <integer expression>}
Scope: Local

Microsoft relies on PE (portable executable) header flags to allow an application to indicate compatiblity with OS services or request advanced OS services. These directives provide powerful options for tuning your applications on high-end NT systems.

These directives allow you to set flag bits in the PE file header Characteristics field and PE file optional header DLLCharacteristics field, respectively. Most of the Characteristics flags, set using $SetPEFlags, are specifically for object files and libraries. DLLCharacteristics, set using $SetPEOptFlags, are flags that describe when to call a DLL's entry point.

The <integer expression> in these directives can include Delphi constant identifiers, such as the IMAGE_FILE_xxxx constants defined in Windows.pas. Multiple constants should be OR'd together."

So it (and the related SetOptPEFlags directive) is a way of informing the OS about the certain aspects of the application. For instance, it can be used to tell the OS that it is ready and compatible with getting access to virtual memory addresses above the 2 GB boundary. Lets look at the IMAGE_FILE constants defined in the Windows.pas unit.

const
  { Relocation info stripped from file. }
  IMAGE_FILE_RELOCS_STRIPPED               = $0001;
  { File is executable  (i.e. no unresolved externel references)}
  IMAGE_FILE_EXECUTABLE_IMAGE              = $0002;
  { Line nunbers stripped from file. }
  IMAGE_FILE_LINE_NUMS_STRIPPED            = $0004;
  { Local symbols stripped from file. }
  IMAGE_FILE_LOCAL_SYMS_STRIPPED           = $0008;
  { Agressively trim working set }
  IMAGE_FILE_AGGRESIVE_WS_TRIM             = $0010;
  { App can handle >2gb addresses }
  IMAGE_FILE_LARGE_ADDRESS_AWARE           = $0020;
  { Bytes of machine word are reversed. }
  IMAGE_FILE_BYTES_REVERSED_LO             = $0080;
  { 32 bit word machine. }
  IMAGE_FILE_32BIT_MACHINE                 = $0100;
  { Debugging info stripped from file in .DBG file }
  IMAGE_FILE_DEBUG_STRIPPED                = $0200;
  { If Image is on removable media, copy and run from the swap file}
  IMAGE_FILE_REMOVABLE_RUN_FROM_SWAP       = $0400;
  { If Image is on Net, copy and run from the swap file. }
  IMAGE_FILE_NET_RUN_FROM_SWAP             = $0800;
  { System File. }
  IMAGE_FILE_SYSTEM                        = $1000;
  { File is a DLL. }
  IMAGE_FILE_DLL                           = $2000;
  { File should only be run on a UP machine }
  IMAGE_FILE_UP_SYSTEM_ONLY                = $4000;
  { Bytes of machine word are reversed. }
  IMAGE_FILE_BYTES_REVERSED_HI             = $8000;

That's a lot of values, but for the purposes of this article, we're really only interested in the first one.

const
  { Relocation info stripped from file. }
  IMAGE_FILE_RELOCS_STRIPPED               = $0001;

That's where our magic number 1 comes from. If we add Windows to the project uses clause, and move the SetPEFlags directive below the uses clause, we can use this more self-documenting directive.

{$SetPEFlags IMAGE_FILE_RELOCS_STRIPPED}

So this tells the OS that the file does not contain any relocation information, and thus it cannot be rebased at runtime. In addition, the Delphi compiler now also picks up on this information and uses it as an instruction to do as you say - leaving out reloc information from the generated EXE file!

Note: The compiler allows using the same directive to remove reloc info from packages and DLLs, but this is not a recommended practice. It will generate slightly smaller DLLs, but the DLL will fail to load unless it can be loaded at its given base address.

Score: Delphi 7 vs. Delphi 2006: 0 - 1 ;)

ODC#2: 1st Oslo Delphi Club event was great!

Unbelievable, but true - only two weeks after the Oslo Delphi Club was created we have 48 members and counting! And on Wednesday September 20th we gathered not less than 21 members to the first ever meeting at the local Peppe's Pizza joint to share good food, beverages, background stories and enthusiasm of Delphi, DevCo and the Oslo Delphi Club.

The room I had booked was packed to the last seat. After getting our first cold glass in hand, we took turns telling about ourselves, our Delphi/ Turbo/Borland Pascal background, our employers/companies, our projects and our ideas, hopes and possible contribution to the future our new-founded club.

Members varied greatly in age, experience and project types, but in common we all had the love in a product and phenomena as splendid as Delphi. We raved about the ingeniousness of a development tool that has followers who will select a product based on the language (read: Delphi) in which it is programmed.

I think that this enthusiasm caters well for the future of Delphi in general and the Oslo Delphi Club in general. If you haven't already, and you're close to Oslo, Norway - be sure to sign up as a member here.

ODC#1: First ever Oslo Delphi Club meeting!

The Oslo Delphi Meetup group has now been created and the first meeting is scheduled to be Wednesday September 20th, at a local pizza place. If you are a Delphi programmer situated in or close to Oslo, make sure you join the group and signup for the meeting.

 

PS. IMO, the Oslo Delphi group hasn't finalized its name yet. The archetypical English name that meetup.com uses is The Oslo Delphi Meetup Group. The current Norwegian name is the awkward "Brukergruppen for Delphi i Oslo" (literally "The user group for Delphi in Oslo) and I'm not satisfied with it. Perhaps "Oslo Delphi Club" could be a swifter, more natural name - it works in both Norwegian and English. We'll probably discuss it at the first meeting - be there if you can!

DN4DP#2: Protecting your privates

This post continues the series of The Delphi Language Chapter teasers from Jon Shemitz’ .NET 2.0 for Delphi Programmers book. Last time we learned about the new kinds of class members that have become available; class fields, class static methods, class properties and class constructors. This time we will look at the new class member visibility specifiers that are available, abstract classes and final methods.

Note that I do not get any royalties from the book and I highly recommend that you get your own copy – for instance at Amazon.

"Protecting your privates

Native Delphi already had four class member visibility levels; private, protected, public and published[1]. One quirk with these is that private and protected members are fully visible to all the code in the unit they are declared in[2], not just the class they are part of (almost like an implicit version of the C++ friend concept). To match .NET's concept of truly private and protected, two new access levels named strict private and strict protected, were introduced. The AppendixDfn\PrivateParts project demonstrates this

type
  TFoo = class
  strict private
    FCantTouchMe: integer;
    FAnyOneAndDelphiRTTI: integer;
  private
    FClassAndUnit: integer;
  strict protected
    FClassAndDescendants: integer;
  protected
    FClassDescendantsAndUnit: integer;
  public
    FAnyOne: integer;
    constructor Create(Report: boolean = True);
  published
    property AnyOneAndDelphiRTTI: integer read FAnyOneAndDelphiRTTI 
  write FAnyOneAndDelphiRTTI;
  end;

Delphi classes can now be explicitly sealed and abstract. The syntax here has the mildly surprising order class sealed and class abstract. The main reason for this is to avoid reserving more language keywords than absolutely necessary – sealed and abstract are directives that only have special meaning after the class reserved word. This means that existing code that already use these identifiers will not break. A sealed class cannot be inherited from and an abstract class cannot be instantiated (even if it does not contain any abstract methods).

Finally, a virtual method that you override can now be marked final, preventing derived classes from overriding that method.

  TAbstractClass = class abstract
  public
    procedure Bar; virtual;
  end;

  TSealedClass = class sealed(TAbstractClass)
  public
    procedure Bar; override;
  end;

  TFinalMethodClass = class(TAbstractClass)
  public
    procedure Bar; override; final;
  end;

---

[1] There is also the obsolete automated section from Delphi 2 – but that is not supported in .NET.

[2] One native Delphi trick is to declare a local descendant of a class in the current unit. Then you hard-cast an object instance into this local class. Now you have access to all the protected members of the object. This hack is so common that the .NET compiler has special logic to handle it too. It does work as long as the original class code is in the same assembly as the hacking code. This is because Delphi’s protected access maps to the .NET family or assembly access level. Similarly, Delphi’s private access maps to the .NET assembly access level, but the Delphi compiler itself enforces the cross-unit privateness."

[Note: This text differs slightly from the final printed version]